Multi-modal fusion verification method, device and equipment for business license verification

Abstract

The application discloses a multi-modal fusion verification method, device and equipment for business license verification. The method comprises the following steps: extracting text information of a business license image to be verified, and performing preliminary verification based on the text information; in response to the preliminary verification passing, the following steps are performed: outputting a first feature vector based on the image features of the business license image, the first feature vector being used to represent the visual features of the business license image; identifying a two-dimensional code in the business license image, obtaining a uniform resource locator by analysis, performing multi-dimensional risk detection based on the uniform resource locator, and constructing a second feature vector according to the multi-dimensional risk detection result; analyzing the editing traces, creation and modification time of the business license image, and the geographical position when the business license is taken based on the metadata of the business license image, and constructing a third feature vector based on the analysis result. The application can improve the accuracy of tampered image detection.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and also to image verification and data processing technology, and in particular to a multimodal fusion verification method, apparatus, equipment, program product and storage medium for business license verification. Background Technology

[0002] In the recruitment industry, to verify the identity of a company and allow it to post job information, the authenticity of the business license image provided by the company must be verified. Currently, mainstream methods in this field often rely on single-modal features, such as extracting text information solely through OCR and comparing it with a public database, or using only general image tampering detection algorithms to analyze pixel-level anomalies. These single-feature verification methods are significantly insufficient in recognizing visually imperceptible tampering performed using professional image editing tools (such as Photoshop), because forgers can easily obtain genuine business license information to forge content while erasing visible traces of image tampering. Furthermore, even when existing technologies attempt to combine multiple features, their fusion methods are often limited to static combinations using simple "AND / OR" logic rules. This results in low system accuracy when faced with complex situations where features contradict each other (e.g., correct text information but image showing signs of manipulation) or where only partial areas have been tampered with. Summary of the Invention

[0003] In view of this, embodiments of this application provide a multimodal fusion verification method, apparatus, electronic device, computer-readable storage medium, and computer program product for business license verification, solving at least one technical problem.

[0004] This application provides a multimodal fusion verification method for business license verification, comprising: extracting text information from a business license image to be verified, and performing preliminary verification based on the text information; in response to the preliminary verification being passed, performing the following multiple steps: outputting a first feature vector based on the image features of the business license image, the first feature vector being used to characterize the visual features of the business license image; identifying a QR code in the business license image, parsing it to obtain a Uniform Resource Locator (URL), performing multidimensional risk detection based on the URL, and constructing a second feature vector based on the multidimensional risk detection results; analyzing the editing traces, creation and modification times, and geographical location when the business license was photographed based on the metadata of the business license image, and constructing a third feature vector based on the analysis results; performing weighted fusion of the first feature vector, the second feature vector, and the third feature vector to obtain a fused feature vector; inputting the fused feature vector into a preset classification model to obtain the tampering probability of the business license image; and outputting a verification result based on the comparison result of the tampering probability and a preset threshold.

[0005] Optionally, according to the method of this application embodiment, the step of performing multi-dimensional risk detection based on the Uniform Resource Locator (URL) and constructing a second feature vector based on the multi-dimensional risk detection results includes: parsing the URL to obtain parameter information therein; determining a consistency risk value based on the degree of matching between the parameter information and corresponding information in the text information; determining an reachability risk value based on the response result of a network request initiated to the URL; determining a domain name risk value based on whether the domain name of the URL exists in a preset official domain name whitelist; determining a batch risk value based on the frequency of the same URL content appearing in verified business license images within a preset time window, wherein the batch risk value = min(1.0, number of images with the same URL content / preset batch size threshold); and constructing the second feature vector based on the consistency risk value, the reachability risk value, the domain name risk value, and the batch risk value.

[0006] Optionally, according to the method of this application embodiment, the step of analyzing the editing traces, creation and modification times, and geographical location when the business license image was photographed, and constructing a third feature vector based on the analysis results, includes: analyzing whether there is signature information of image editing software in the metadata of the business license image, and determining software editing feature values ​​based on the signature information analysis results; analyzing whether there are corresponding values ​​between the creation time and modification time in the metadata of the business license image, and determining timeliness anomaly feature values ​​based on the time analysis results; extracting the shooting location information from the metadata of the business license image, and extracting the company registration address from the text information; calculating the distance between the business license shooting location in the shooting location information and the company registration address, and obtaining a geographical location anomaly feature value based on the relationship between the distance and a preset distance threshold; and using the software editing feature value, the timeliness anomaly feature value, and the geographical location anomaly feature value as feature values ​​of the third feature vector to obtain the third feature vector.

[0007] Optionally, according to the method of this application embodiment, the signature information includes Make, XMPToolkit, TIFF tag, and Software metadata, and the software editing feature value is 0 or 1. The analysis process includes: extracting the values ​​of each metadata in the signature information according to a preset extraction order, and returning the value as the result if there is a value; if there is a value of metadata in the returned result, it is determined that the business license image has editing software traces, and the software editing feature value is 1.

[0008] Optionally, according to the method of this application embodiment, the distance between the business license shooting location in the shooting location information and the company registration address is calculated, and a geographic location anomaly feature value is obtained based on the relationship between the distance and a preset distance threshold, including: converting the company registration address into latitude and longitude coordinates; calculating the spherical distance D between the latitude and longitude coordinates of the business license shooting location in the shooting location information and the latitude and longitude coordinates of the registration address; if the distance D ≤ threshold T, then the geographic location anomaly feature value is 0; if the distance D > threshold T, then the geographic location anomaly feature value = min(1.0, (DT) / (K×T)), where K is a constant greater than 1.

[0009] Optionally, according to the method of this application embodiment, the first feature vector, the second feature vector, and the third feature vector are weighted and fused to obtain a fused feature vector, including: projecting the first feature vector, the second feature vector, and the third feature vector to the same feature dimension through a linear transformation layer; generating weight values ​​for the projected first feature vector, the second feature vector, and the third feature vector based on the projected first feature vector, the second feature vector, and the third feature vector through a cross-attention mechanism and a decision neural network; and performing a weighted summation of the three projected feature vectors according to the weight values ​​to obtain the fused feature vector.

[0010] Optionally, according to the method of this application embodiment, the method further includes an offline classification model iteration step, which includes: monitoring classification model iteration trigger conditions, the trigger conditions including at least one of the following: the number of newly accumulated manually labeled positive and negative samples reaches a preset number, the performance index of the online classification model on the validation set is continuously lower than a preset threshold, and a preset periodic time point is reached; in response to any of the trigger conditions being met, a training process is started, the training process including: acquiring labeled positive sample images and negative sample images; performing a multimodal feature extraction step and a dynamic weighted fusion step on the positive sample images and the negative sample images to construct a training dataset; retraining the classification model using the training dataset, and selecting the optimal model based on the comprehensive performance index of the independent validation set; and deploying the optimal model as a new online classification model.

[0011] Optionally, according to the method of this application embodiment, the preliminary verification includes: comparing the key fields in the text information with the corresponding enterprise registration information queried through official sources online; if the comparison is inconsistent, the preliminary verification is determined to be unsuccessful, and a verification failure conclusion is directly output.

[0012] Optionally, the method according to the embodiments of this application further includes: generating metadata confidence feature values ​​based on the metadata of the business license image, wherein the metadata confidence feature values ​​are used to characterize the editing traces, creation and modification time of the business license image, and the reliability of the geographical location when the business license was photographed; performing weighted fusion of the first feature vector, the second feature vector, and the third feature vector to obtain a fused feature vector, including: generating weight values ​​for the first feature vector, the second feature vector, and the third feature vector based on the first feature vector, the second feature vector, the third feature vector, and the metadata confidence feature values, respectively, through a cross-attention mechanism and a decision neural network, wherein the weight value of the third feature vector in the attention weights is positively correlated with the metadata confidence feature values; and constructing a fused feature vector based on the weight values ​​and the first feature vector, the second feature vector, and the third feature vector.

[0013] This application proposes a multimodal fusion verification device for business license verification, comprising: a preliminary verification module for extracting text information from a business license image to be verified and performing preliminary verification based on the text information; a first feature vector construction module for outputting a first feature vector based on the image features of the business license image in response to the preliminary verification being passed, the first feature vector being used to characterize the visual features of the business license image; and a second feature vector construction module for recognizing a QR code in the business license image, parsing it to obtain a Uniform Resource Locator (URL), performing multidimensional risk detection based on the URL, and determining the multidimensional risk detection result. The system comprises: a second feature vector construction module; a third feature vector construction module, used to analyze the editing traces, creation and modification time, and geographical location when the business license image was photographed based on the metadata of the business license image, and construct a third feature vector based on the analysis results; a weighted fusion module, used to perform weighted fusion of the first feature vector, the second feature vector, and the third feature vector to obtain a fused feature vector; an input module, used to input the fused feature vector into a preset classification model to obtain the tampering probability of the business license image; and an output module, used to output a verification result based on the comparison result of the tampering probability and a preset threshold.

[0014] This application provides an electronic device, which includes a processor and a memory storing computer program instructions; the electronic device executes the computer program instructions to implement the method described above.

[0015] This application provides a computer program product, which includes computer program instructions that, when executed, implement the method described above.

[0016] This application provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the method described above.

[0017] This application embodiment achieves weighted decision-making by fusing image visual features, QR code hidden information risk features, and metadata features, effectively overcoming the limitations of relying on a single feature or simple rule combination, and improving the detection accuracy of professional image tampering. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings of the embodiments of this application will be briefly described below.

[0019] Figure 1 This is a schematic diagram of the system architecture of an embodiment of this application.

[0020] Figure 2 This is a flowchart of a multimodal fusion verification method for business license verification according to an embodiment of this application.

[0021] Figure 3 This is a flowchart illustrating the process of a multimodal fusion verification method for business license verification according to an embodiment of this application.

[0022] Figure 4 This is a flowchart of the training and updating process of the classification module in an embodiment of this application.

[0023] Figure 5 This is a schematic structural block diagram of a multimodal fusion verification device for business license verification according to an embodiment of this application.

[0024] Figure 6 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0025] The principles and spirit of this application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided to make the principles and spirit of this application clearer and more thorough. The exemplary embodiments provided herein are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application.

[0026] Embodiments of this application relate to terminal devices and / or servers. Implementations of this application can be a system, terminal, device, method, computer-readable storage medium, or computer program product, and can be specifically implemented as entirely hardware, entirely software, or a combination of hardware and software. Figure 1This diagram illustrates a system architecture according to an embodiment of the present application, including a terminal device 102 and a server 104. The terminal device 102 may include at least one of the following: a smartphone, tablet computer, laptop computer, desktop computer, smart TV, various wearable devices, augmented reality (AR) devices, virtual reality (VR) devices, etc. A client application, such as an app, mini-program, or browser-based client, can be installed on the terminal device 102. Users can input commands through the client, and the terminal device 102 can send request information containing the commands to the server 104. Upon receiving the request information, the server 104 performs corresponding processing and returns the processing result information to the terminal device 102. The server 104 may be a local server or a cloud server, and may be a single server or a server cluster, etc.

[0027] In this document, the terms "first," "second," "third," etc., are used only to distinguish one entity (or operation) from another in textual description, and do not require or imply any sequential order between these entities (or operations).

[0028] Figure 2 A flowchart illustrating a multimodal fusion verification method for business license verification according to an embodiment of this application is shown. The method includes the following steps: S101: Extract the text information from the business license image to be verified, and perform preliminary verification based on the text information; S102: In response to the initial verification being passed, the following steps are performed: S103: Output the first feature vector based on the image features of the business license image. The first feature vector is used to characterize the visual features of the business license image. S104: Identify the QR code in the business license image, parse it to obtain the Uniform Resource Locator (URL), perform multidimensional risk detection based on the URL, and construct a second feature vector based on the multidimensional risk detection results; S105: Based on the metadata of the business license image, analyze the editing traces, creation and modification time of the business license image, as well as the geographical location when the business license was taken, and construct a third feature vector based on the analysis results; S106: Weighted fusion of the first feature vector, the second feature vector, and the third feature vector is performed to obtain the fused feature vector; S107: Input the fused feature vector into the preset classification model to obtain the tampering probability of the business license image; S108: Output the verification result based on the comparison between the tampering probability and the preset threshold.

[0029] In this embodiment of the application, it is necessary to perform preliminary text information extraction and verification on the business license image submitted by the user. Specifically, key fields such as the unified social credit code, name, and address of the enterprise in the image can be obtained through optical character recognition technology and compared online with publicly available information data. If a mismatch is found in the key information, the process is terminated directly and the verification failure is returned, thereby efficiently filtering out low-quality or obviously forged inputs.

[0030] After the initial verification is passed, the system initiates multimodal feature extraction and analysis, including capturing the overall visual pattern and potential tampering traces of the image at the pixel level to construct the first feature vector; constructing the second feature vector by parsing the QR code in the business license image and evaluating the performance of the URL it points to on multidimensional risk indicators; and constructing the third feature vector reflecting the image generation and modification history by analyzing the editing software signature, file time information, and shooting geographical location and registration address in the metadata of the business license image.

[0031] Specifically, for the first feature vector, the system can use a deep convolutional neural network model to encode the visual features of the business license image itself, compressing and transforming the high-dimensional pixel information of the entire image into a first feature vector with a fixed dimension. This vector can characterize the overall visual style, texture statistical characteristics, and potential, imperceptible local tampering traces of the image, such as unnatural seams, abnormal color channels, or inconsistent resolution.

[0032] For the second feature vector, the location and decoding are first performed to obtain the Uniform Resource Locator (URL) carried in the QR code of the business license image. Then, a series of risk detections are carried out around the URL. These detection dimensions include, but are not limited to: the consistency between the unified social credit code carried in the URL parameter and the result identified by the aforementioned optical character recognition technology; the network reachability of the URL address itself; whether the domain name to which the URL belongs is within the preset official query platform whitelist; and the frequency of the same URL appearing in batch verification requests within a specific time window. Each of the above risk dimensions is quantified into a risk score, and they are combined to form a second feature vector that comprehensively reflects the credibility and abnormal patterns of the QR code information.

[0033] To construct the third feature vector, the system needs to parse the EXIF ​​and other metadata tags of the business license image to identify whether the image was generated or modified by specific image editing software. Furthermore, the system needs to analyze the logical relationship and timeliness between the creation time and last modification time of the business license image to determine if the image was recently created. Additionally, the system needs to extract the possible geographical location of the business license photo and compare it with the company's registered address to determine if the distance between the two meets preset conditions. Finally, the analysis results of various scenarios are used as feature values ​​to construct the third feature vector.

[0034] In this embodiment, after obtaining the first feature vector, the second feature vector, and the third feature vector, the feature vectors need to be weighted and fused to obtain a fused feature vector. The weights of the first, second, and third feature vectors can vary for different business license images. In some embodiments of this application, optionally, the data related to the business license image used in constructing each feature vector can be input into a preset neural network. This preset neural network can dynamically generate attention weights corresponding to each feature vector based on the reliability of the input data. In this way, the neural network can adaptively adjust based on the saliency and credibility of various features during feature vector generation, thereby obtaining different weights for different business license images. For example, when encountering a high-quality visual stitching forgery, the image feature vector may respond weakly, but traces of editing software or abnormal time differences left in the metadata may cause the third feature vector to exhibit a high weight. Simultaneously, if the forger reuses a legitimate QR code, although the consistency risk in the second feature vector is low, the system detects that the QR code appears in multiple business license images, thus posing a high risk of batch use. Therefore, in this case, the system may assign a higher weight to the second feature vector.

[0035] Therefore, this embodiment of the application can obtain a fused feature vector that better suits the various data conditions by adjusting the weights of each feature vector. Finally, the fused feature vector is input into a pre-trained classification model. Based on the fused multidimensional evidence, the model can calculate the probability value that the business license image to be verified is a tampered image. The system then automatically outputs the final verification conclusion by comparing this probability value with a preset decision threshold.

[0036] This application embodiment achieves weighted decision-making by fusing image visual features, QR code hidden information risk features, and metadata features, effectively overcoming the limitations of relying on a single feature or simple rule combination, and improving the detection accuracy of professional image tampering.

[0037] According to some embodiments of this application, optionally, multi-dimensional risk detection is performed based on Uniform Resource Locators (URLs), and a second feature vector is constructed based on the multi-dimensional risk detection results, including: Parse the Uniform Resource Locator (URL) to obtain its parameter information; The consistency risk value is determined based on the degree of matching between the parameter information and the corresponding information in the text information. The reachability risk value is determined based on the response result of the network request initiated to the Uniform Resource Locator; The risk value of a domain is determined by whether it exists in the pre-defined official domain whitelist based on the Uniform Resource Locator (URL). The batch risk value is determined based on the frequency of the same Uniform Resource Locator (URL) content appearing in the verified business license images within a preset time window. The batch risk value is min(1.0, number of images with the same URL content / preset batch size threshold). A second feature vector is constructed based on consistency risk value, reachability risk value, domain name risk value, and batch risk value.

[0038] In this embodiment, a layered and multi-dimensional in-depth risk analysis is required on the Uniform Resource Locator (URL) carried by the QR code embedded in the business license image. The system first parses the URL to extract key identification information, such as the Unified Social Credit Code, carried in its query parameters. Subsequently, the system initiates a series of structured risk detection logics: the first detection focuses on parameter consistency. By precisely comparing the parameter values ​​parsed from the URL with the corresponding information identified by OCR from the image text, a consistency risk value is quantified. This value directly reflects whether the information on the QR code surface and the visible content of the image have been tampered with or replaced. In some embodiments of this application, optionally, if the results are consistent, the consistency risk value is 0; if inconsistent, it is 1.

[0039] In this application, the system also needs to initiate a network request to the URL address, determine its network reachability based on whether a valid response can be obtained within a set timeout period (e.g., 2 seconds), and generate a reachability risk value. The purpose is to identify QR codes that point to fictitious or invalid web pages. In some embodiments of this application, optionally, if reachable, the reachability risk value is 0, and if unreachable, it is 1.

[0040] In this embodiment, the system also detects domain trustworthiness by checking whether the domain of the URL exists in a pre-maintained whitelist of an official authoritative query platform, and outputs a domain risk value accordingly to filter out links pointing to unofficial or suspicious sites. Optionally, in some embodiments of this application, the domain risk value is 0 if it is in the whitelist, and 1 if it is not.

[0041] In addition, this embodiment of the application will also count the frequency of the same URL content in all images to be verified within a preset time window, and calculate a batch risk value based on a set batch size threshold. This value can effectively reveal whether someone is using the same template for large-scale forgery. Batch risk value = min(1.0, number of images with the same Uniform Resource Locator content / preset batch size threshold). Finally, the system will use these four dimensions of risk values ​​as feature values ​​to form a comprehensive second feature vector, thereby completing a comprehensive and detailed risk assessment of the QR code as a hidden information channel.

[0042] This application's embodiments construct a robust framework for multi-level credibility assessment of QR code information through multiple complementary risk dimensions. Consistency detection ensures the logical unity of the embedded data and visual content of the QR code, preventing information grafting forgery; accessibility detection verifies the authentic existence of the digital resources behind the QR code, eliminating interference from fake links; whitelist domain detection anchors the official authority of the information from its source, significantly reducing the risk of counterfeit websites; and batch frequency detection, from the perspective of business behavior patterns, can keenly detect forged evidence with identical content generated through industrial means. This multi-indicator joint evaluation mechanism greatly enhances the system's ability to identify complex forgery strategies. While the risk of a single dimension may be deliberately avoided by forgers, the difficulty of simultaneously deceiving all dimensions and maintaining a natural state increases exponentially, thereby significantly improving the accuracy of the overall verification system.

[0043] According to some embodiments of this application, optionally, the editing traces, creation and modification times of the business license image, and the geographical location when the business license was photographed are analyzed, and a third feature vector is constructed based on the analysis results, including: Analyze the metadata of the business license image to determine if there is signature information of the image editing software, and determine the software editing feature value based on the analysis results of the signature information; Analyze the metadata of the business license image to see if there are corresponding values ​​for the creation time and modification time, and determine the timeliness anomaly characteristic value based on the time analysis results; Extract the shooting location information from the metadata of the business license image, and extract the company registration address from the text information; Calculate the distance between the location where the business license was photographed and the company's registered address in the location information, and obtain the geographical location anomaly feature value based on the relationship between the distance and a preset distance threshold; The software editing feature value, the timeliness anomaly feature value, and the geographical location anomaly feature value are used as the feature values ​​of the third feature vector to obtain the third feature vector.

[0044] In this application embodiment, the metadata of the business license image file needs to be checked for specific signature information left by professional image editing software such as "Photoshop" when saving, thereby generating a software editing feature value that directly indicates whether the file has been processed by such a powerful tampering tool. In some embodiments of this application, optionally, if editing software is detected, the software editing feature value is 1; otherwise, it is 0. In addition, the system will also accurately read and compare the file creation time and last modification time recorded in the metadata, analyzing whether the time difference between the two conforms to the logic of normal business processes. For example, if an image claimed to be an original scan has a modification time much later than its creation time and occurred recently, it may indicate that it has been edited afterward, and the system will generate a timeliness anomaly feature value accordingly. In some embodiments of this application, optionally, if the modification time is recent, the timeliness anomaly feature value is 1; otherwise, it is 0.

[0045] Furthermore, in this embodiment, it is also necessary to reconstruct the physical scene at the time of shooting from the metadata, extract potential business license shooting location information, such as the latitude and longitude of the shooting location, and obtain the company's legal registered address from the text recognized by OCR. Geocoding services are then used to convert both into coordinates and calculate their distance. If the shooting location is far from the company's registered address—for example, a small business registered locally might have its business license image shown as being taken thousands of miles away—a high geolocation anomaly feature value will be generated. In some embodiments of this application, the geolocation anomaly feature value is a value between 0 and 1. Finally, these three feature values, mined from tool traces, temporal logic, and spatial logic levels respectively, combine to form the third feature vector.

[0046] The three feature values ​​proposed in this application complement each other from a construction perspective, jointly constructing a three-dimensional anti-counterfeiting dimension. Even if counterfeiters successfully imitate the visual appearance and QR code content of the business license, they can easily leave traces at the metadata level that are difficult to completely conceal and violate normal business practice logic, thereby significantly improving the system's recognition rate of counterfeit business license images.

[0047] According to some embodiments of this application, optionally, the signature information includes a Make field, an XMPToolkit field, a TIFF tag, software metadata, and a software editing feature value of 0 or 1. The analysis process includes: Extract the values ​​of each metadata element in the signature information according to the preset extraction order, and return the value as the result if there is a value. If the returned result contains metadata values, it indicates that the business license image has been edited by software, with the software editing feature value being 1.

[0048] In this embodiment, the signature information includes at least four fields written or modified by various image editing software: Make (device manufacturer information), XMPToolkit (Adobe Extended Metadata Platform Toolkit identifier), TIFF tag (image file format related metadata), and Software (software name). In this embodiment, when the system performs analysis, it can iteratively extract the aforementioned metadata according to a preset, designed fixed priority order (e.g., sequentially attempting to extract the values ​​of Make, XMPToolkit, TIFF tag, and Software). This sequential logic ensures the efficiency of the investigation; the system starts checking from the field most likely to carry the editing software identifier. Once a non-empty value is successfully extracted from any field, the extraction of subsequent fields is immediately terminated, and the extraction result is taken as the final output of this investigation. Since different editing software or processing flows may have different positions and habits for writing signature information in different image formats, the fixed-order traversal method can capture at least one valid trace with a high success rate. Ultimately, as long as any metadata item is successfully extracted with a valid value during the entire traversal process, regardless of whether the value is "Adobe Photoshop CS6" or other editing software name, the system will consistently determine that the business license image has editing software traces and quantify the software editing feature value as 1; conversely, if all specified fields are empty, the feature value will be 0.

[0049] This application's embodiments significantly improve the detection coverage of traces left by various image editing software (whether professional-grade Photoshop or other common tools) by employing composite signature information containing multiple types and sources of tags. This avoids the risk of missing specific software tampering behavior due to reliance on a single tag, enhancing the system's versatility. Secondly, the preset fixed priority extraction order optimizes the execution efficiency of the detection process, enabling the system to make effective judgments within the shortest path and reducing unnecessary full scanning. This is particularly important for online verification scenarios that require processing massive amounts of images.

[0050] According to some embodiments of this application, optionally, the distance between the location where the business license was photographed and the company's registered address in the location information is calculated, and based on the relationship between the distance and a preset distance threshold, a geographic location anomaly feature value is obtained, including: Convert the company's registered address to latitude and longitude coordinates; Calculate the spherical distance D between the latitude and longitude coordinates of the business license shooting location and the registered address in the shooting location information; If distance D ≤ threshold T, then the geographic location anomaly feature value is 0; if distance D > threshold T, then the geographic location anomaly feature value = min(1.0, (DT) / (K×T)), where K is a constant greater than 1.

[0051] In this embodiment, to calculate the distance between the location where the business license was photographed and the company's registered address, the text-based registered address needs to be converted into standard latitude and longitude coordinates using a reliable geocoding service. Simultaneously, the latitude and longitude coordinates recorded at the time of photographing are extracted from the image metadata, thus unifying the two locations under the same geospatial reference system. Subsequently, the system calculates the spherical distance D between these two points. This calculation method, compared to a simple straight-line distance, better reflects the actual geographical conditions of the Earth's surface, ensuring the accuracy of the measurement.

[0052] After obtaining the distance D, if the actual distance D does not exceed the set distance threshold T (which may be set based on the city radius or a reasonable office radiation range), the shooting location is considered to be within a reasonable range, and the feature value is directly set to 0, indicating no anomaly. If the distance D exceeds the threshold T, it indicates a geographical inconsistency, and the geographical location anomaly feature value can be calculated using the formula min(1.0,(DT) / (K×T)). The geographical location anomaly feature value of this application will smoothly increase from 0 as the excess distance (DT) increases, and its growth rate is controlled by the denominator K×T, where K is a constant greater than 1 (e.g., 3), which acts to "stretch" the buffer between anomaly judgments. This means that when the distance D slightly exceeds the threshold T, the feature value will be a small value (e.g., 0.1 or 0.2), and only when the distance D far exceeds the threshold (reaching T + K×T or even further) will the feature value approach or equal to 1.

[0053] This application's embodiment transforms the physical consistency between the business license photo location and the registered address into a quantifiable feature, enriching the verification dimensions. It constructs a non-linear quantization function by setting a distance threshold T and a constant K, allowing the feature value to smoothly increase with the distance exceeding the threshold, rather than a binary judgment of either / or. This enhances robustness to latitude and longitude errors or situations involving multiple legitimate business locations, effectively focusing on significant anomalies. The continuous output values ​​(0~1) provide a more refined basis for subsequent multimodal fusion, thereby improving the accuracy and overall reliability of the system's comprehensive reasoning.

[0054] According to some embodiments of this application, optionally, the first feature vector, the second feature vector, and the third feature vector are weighted and fused to obtain a fused feature vector, including: The first feature vector, the second feature vector, and the third feature vector are projected onto the same feature dimension through a linear transformation layer; Based on the first, second, and third feature vectors after projection, the weight values ​​of the first, second, and third feature vectors after projection are generated through a cross-attention mechanism and a decision neural network, respectively. Based on the weight values, the three projected feature vectors are weighted and summed to obtain the fused feature vector.

[0055] In this embodiment, the first feature vector is typically a high-dimensional array (e.g., 512-dimensional or 2048-dimensional), extracted by a deep convolutional neural network (such as ResNet), encoding the global visual features and local texture patterns of the business license image. The second feature vector is a multi-dimensional vector (e.g., 4-dimensional) composed of multiple risk scores (e.g., consistency risk, accessibility risk, etc.), quantifying the credibility and anomaly patterns of the digital information associated with the QR code in the image. The third feature vector is an array that comprehensively reflects the logical anomalies of metadata (e.g., containing three feature values: software editing, time anomaly, and geographical distance anomaly). These three vectors, with completely different dimensions, scales, and physical meanings, are each projected onto a predefined, uniform feature dimension (e.g., 256-dimensional) through a dedicated linear transformation layer (usually a fully connected layer).

[0056] The weight generation process is illustrated below using the visual feature vector (V), QR code feature vector (Q), and metadata feature vector (M) projected onto a 256-dimensional unified feature space as examples: First, the attention cross-mapping mechanism generates a context enhancement vector based on the three input feature vectors. The goal of this stage is to enable each modality's features to actively "inquire" and integrate information from other modalities. Specifically, it generates its context enhancement vector Context from the metadata feature vector M. M The process is illustrated using an example. First, M is used as the "Query," and the combined feature [V; Q] formed by concatenating V and Q serves as the source of the "Key" and "Value." The system uses three sets of trainable parameter matrices ( , , The original vectors are linearly transformed to obtain the query vector. = M × key vector = [V; Q] × and value vector = [V; Q] × .

[0057] Next, the relevance score is calculated. This is done by calculating the query vector. With key vector It is implemented by the dot product of each component vector, which essentially measures the correlation strength between the current representation of M and each part of the information in [V; Q]. For example, The portion representing "the presence of Photoshop editing traces" in the V section may have a high positive correlation score with the sub-feature dimension in V describing "extraordinarily smooth image edges," while its score with the sub-feature dimension in Q describing "URL parameter consistency" may be close to zero. These raw scores are scaled and then normalized using a Softmax function, transforming them into a set of weighted coefficients. This set of weighted coefficients clearly reveals which information from V and Q is considered crucial in this specific verification to respond to M's "question."

[0058] Finally, use this set of normalized weights to pair the value vector. Perform a weighted summation to generate a context-enhanced vector. Context enhancement vector It maintains dimensionality consistent with the original projection vector (e.g., 256 dimensions). Context-enhanced vector It is an aggregate of information most relevant to the current metadata features, filtered and focused from both visual and QR code perspectives. For example, it can be interpreted that subtle anomalies in certain texture statistics exhibited by visual features partially support the inference of professional editing; while QR code features do not provide valuable information on this issue. Similarly, the system will generate queries in parallel using V and key / value pairs using [Q; M]. Generate a Context using Q as the query and [V; M] as the key / value pair. Q Thus, we have obtained three original projection vectors (V, Q, M) and three context-enhanced vectors rich in cross-modal relationships generated after cross-attention interactions. , , ).

[0059] Next, the concatenation of the above six vectors (a high-dimensional comprehensive evidence representation, for example, 6 × 256 = 1536 dimensions) is input into a decision neural network (usually a multilayer perceptron, MLP). The input vectors encompass all the original evidence and their intricate interrelationships. The neural network's analysis is achieved through its hierarchical nonlinear transformations. The first layer (fully connected layer) might be responsible for extracting higher-order, abstract combinatorial features from the 1536-dimensional original information. For example, identifying a specific pattern combination: "When the input simultaneously contains 'M indicates a high risk of software editing', '..." This risk is supported by specific texture anomalies in V, and ' When the risk of Q is shown to be irrelevant to the context, it constitutes an intermediate feature. After introducing nonlinearity through an activation function (such as ReLU), subsequent network layers perform deeper combinations and reasoning based on these abstract intermediate features to obtain the final decision logic.

[0060] Ultimately, the output layer (usually a fully connected layer followed by a Softmax function) maps the complex inference results within the network to three specific scalar values: weights v, q, and m. The Softmax function ensures that the sum of these three values ​​is 1, transforming them into a probabilistic weight allocation. In this process, the neural network essentially evaluates not only the strength of each feature itself, but also the credibility and contribution of each feature within the overall evidence network formed by other features. Taking the aforementioned forged image as an example, the neural network's analysis logic might be: "Although V itself appears normal,..." This reveals that V failed to refute M's accusations, therefore V's credibility should be lowered; M's accusations are not only strong in themselves, but also supported by... The evidence from V in the middle confirms that M should receive the highest weight; although Q itself carries risk, its risk is not highly correlated with the core contradiction between M and V. and (Since no strong correlation was shown), a moderate weight was assigned. Based on this analysis, the weight values ​​for the first eigenvector, m=0.55, the second eigenvector, v=0.15, and the third eigenvector, q=0.30, were output.

[0061] Finally, in this embodiment, the projected feature vector (V, Q, M) is weighted and summed based on the obtained weight values ​​to obtain the final fused feature vector. This fused feature vector is not a simple mixture of information, but rather it is the result of first uncovering deep relationships between features using an attention mechanism, and then an importance assessment made by a decision neural network based on all evidence and questioning records. This design enables the system to understand features at the relational level, thus providing high detection sensitivity against carefully designed forgeries that attempt to deceive on a single modality, especially those forgeries that leave inconsistencies across multiple modalities that are difficult to completely conceal.

[0062] According to some embodiments of this application, the method may optionally further include an offline classification model iteration step, which includes: The monitoring classification model iteration trigger conditions include at least one of the following: the number of newly accumulated positive and negative samples with manually labeled labels reaches a preset number, the performance index of the online classification model on the validation set is continuously lower than a preset threshold, and a preset periodic time point is reached. In response to any triggering condition being met, the training process is initiated, which includes: Obtain the labeled positive and negative sample images; Perform multimodal feature extraction and dynamic weighted fusion steps on positive and negative sample images to construct a training dataset; The classification model was retrained using the training dataset, and the best model was selected based on the comprehensive performance metrics of the independent validation set. Deploy the optimal model as a new online classification model.

[0063] The system in this application embodiment tracks three key triggering conditions in real time: the number of newly accumulated manually verified and labeled samples, performance indicators such as the accuracy of the current online model on the independent validation set, and preset periodic time points. Once any condition is met, such as when the system detects that the model's judgments are continuously wrong due to the emergence of new forgery methods, or when the periodic update time has expired, the offline training process will be automatically started. This process first collects all recently labeled positive samples (real or tampered images) and negative samples to form a dataset reflecting the latest business distribution. Then, the system fully reuses the multimodal feature extraction (i.e., the extraction process of first, second, and third feature vectors) and dynamic weighted fusion used in online detection to process each image in the dataset, generating a unified fused feature vector and its corresponding label, thus constructing a training dataset with a consistent format. Next, the classification model is retrained using this dataset, and the best-performing new model is selected based on comprehensive performance metrics such as F1-score and AUC on a validation set that was not used in the training. Finally, the system deploys the validated best model online using robust methods such as A / B testing to replace the original model, thereby achieving iterative model updates.

[0064] This application's embodiments transform feedback data generated during online operation (especially manual review results) into training resources, enabling the system to proactively learn and adapt to emerging forgery patterns, thus ensuring long-term effectiveness by keeping detection capabilities and forgery methods evolving in sync.

[0065] According to some embodiments of this application, optionally, preliminary verification includes: The key fields in the text information are compared online with the corresponding enterprise registration information obtained through official sources; If the comparison is inconsistent, the preliminary verification is deemed to have failed, and a verification failure conclusion is output directly.

[0066] After a user uploads an image of a business license, the system first extracts key text fields from the image using OCR technology, such as the unified social credit code and company name. It then immediately initiates an online query request to the official enterprise information disclosure system via a secure interface, comparing the extracted text with authoritative registration records from official sources in real time and with high precision. If any inconsistencies are found in the key information, such as the identified credit code corresponding to another company, the system immediately determines that the image has failed the initial verification, the process terminates directly, and a clear verification failure conclusion is returned, without performing any further complex feature extraction and fusion calculations. Through this process, this application can quickly filter out a large number of inferior, fabricated images based on spliced ​​real information or completely fictitious images with extremely low computational cost, thereby greatly improving the processing efficiency and resource utilization of the entire system.

[0067] Optionally, according to some embodiments of this application, it also includes: Metadata confidence feature values ​​are generated based on the metadata of the business license image. These metadata confidence feature values ​​are used to characterize the editing traces, creation and modification time of the business license image, and the reliability of the geographical location when the business license was taken. The first, second, and third feature vectors are weighted and fused to obtain a fused feature vector, which includes: Based on the first feature vector, the second feature vector, the third feature vector, and the metadata confidence feature value, the weight values ​​of the first feature vector, the second feature vector, and the third feature vector are generated through a cross-attention mechanism and a decision neural network, respectively. Among them, the weight value of the third feature vector in the attention weight is positively correlated with the metadata confidence feature value. A fused feature vector is constructed based on the weight values ​​and the first, second, and third feature vectors.

[0068] While analyzing image metadata and constructing a third feature vector, the system generates a metadata confidence feature value in parallel. This value is used to quantify the credibility and completeness of the metadata itself. For example, it may be calculated based on the degree of missing metadata fields, the logical rationality of timestamps, or the precision of latitude and longitude data.

[0069] During weighted fusion, the system inputs visual feature vectors, QR code risk feature vectors, and metadata content feature vectors into a cross-attention mechanism to obtain enhanced context vectors for each feature vector. Then, the enhanced context vectors and confidence scores are input into a decision neural network. The neural network is trained so that the final weight of the third feature vector (metadata content) is positively correlated with the metadata confidence score. This means that when the confidence assessment report indicates "the metadata quality is very poor and may have been tampered with" (low confidence), the neural network tends to lower the weight of the metadata content feature in the final decision; conversely, when the metadata appears very complete and reliable (high confidence), the weight of its content feature will be relatively increased. Finally, the system performs a weighted fusion of the three feature vectors according to their weights to obtain a fused feature vector.

[0070] The implementation methods and advantages of the embodiments of this application have been described above through multiple examples. The specific processing procedures of the embodiments of this application are described in detail below with reference to specific examples.

[0071] Suppose a meticulously forged image of a "XX Technology Co., Ltd." business license is uploaded to the website's backend. The recruiting company needs to verify the authenticity of the business license image, combined with... Figure 3 As shown, the entire verification process is as follows: Step 1: The user uploads an image of their business license. The system first extracts text using OCR, recognizing key fields such as the Unified Social Credit Code, company name, and address. Then, the system automatically connects to the official interface of the official Enterprise Credit Information Disclosure System and performs a real-time query using the recognized credit code. Upon comparison, if the OCR-recognized "Legal Representative" is found to be "Li Mou," while the official database records "Wang Mou," indicating a mismatch, the system immediately terminates all subsequent complex calculations and directly returns a "Verification Failed" result to the user, along with a message indicating information discrepancy. If the information recognized by the OCR matches the officially published information, the system proceeds to Step 2.

[0072] Step Two: Since the forged image matches the official record in key text information, it enters the multimodal deep analysis process. Specifically, the image is input into a pre-trained ResNet network. Because the forgery is a high-precision stitching of a high-resolution genuine license and a fake legal entity information area generated by Photoshop, the first feature vector output by ResNet globally represents the reasonable texture of the "printed document." However, in the deeper features of the corresponding legal entity information area, subtle and unnatural pixel statistical patterns may be hidden.

[0073] Next, the system identifies the QR code in the image and decodes it to obtain a URL, such as http: / / www.some-fake-site.com / query?uniscid=91370100MA3F123456. Then, a multi-dimensional risk detection is performed, including: Consistency risk: The credit code in the URL parameters matches the OCR result, risk value is 0. Accessibility risk: A request to this URL allows access to the website (possibly a fake query page), risk value is 0. Domain risk: The domain name some-fake-site.com is not on the preset official whitelist (e.g., gsxt.gov.cn), risk value is 1. Batch risk: In the verification batch of the most recent hour, 15 licenses from different companies were found pointing to the same domain name, far exceeding the threshold (e.g., 10 licenses), risk value is min(1.0, 15 / 10)=1.

[0074] In summary, the second feature vector is [0, 0, 1, 1], which strongly suggests that the QR code link carries a high risk of being associated with "unofficial domains" and "mass forgery".

[0075] To obtain the metadata of the business license image, the system parses the EXIF ​​information of the image file. In the Software field, a signature of "Adobe Photoshop 22.0" was found (software editing feature value = 1); the creation time (DateTime) was one year ago, while the modification time (ModifyDate) was three days ago, a significant time difference indicating recent modification (timeliness anomaly feature value = 1); no latitude and longitude information was found (geographical location anomaly feature value = None, or denoted as 0). Therefore, the third feature vector is initially [1, 1, 0].

[0076] At the same time, the system assesses the quality of this "metadata report" itself. Due to the missing key fields (such as latitude and longitude) and the highly suspicious combination of professional software signature and recent modifications, the system judges that the "reliability" of this metadata is questionable and may have been deliberately edited. Therefore, the system generates a low metadata confidence feature value, such as 0.3 (range 0-1, the lower the value, the less reliable the metadata itself).

[0077] Step 3: Dynamic Weighted Fusion and Decision. The three original feature vectors are projected onto the same dimension. This single original feature vector, along with the metadata confidence feature value (0.3), is input into the dynamic weight generation module (which includes an attention crossover mechanism and a decision neural network). The attention crossover mechanism begins to work; for example, the metadata vector [1,1,0] is used as a "query" to query the visual vector and the QR code vector. It finds that the visual vector provides some vague anomaly support in the corresponding region, while the QR code vector is a high-risk signal [0,0,1,1], both of which are related to the "suspicious" conclusion of the metadata to varying degrees. Similar interactions occur across all modalities, constructing a rich joint context.

[0078] The decision neural network comprehensively analyzes all information, paying particular attention to the contradictory yet crucial pattern of "metadata content itself being highly suspicious (vector [1,1,0]), but its source confidence being very low (only 0.3)." Based on pre-trained logic (weights are positively correlated with confidence), the neural network considers the third feature vector to have weak evidence. Therefore, it assigns a lower weight value to the third feature vector (metadata content), for example, a weight m = 0.40. Meanwhile, the QR code vector also receives some weight due to its high risk, with a weight q = 0.50 for the second feature vector, while the visual vector has a very low weight due to its extremely weak presence, with a v = 0.10 for the first feature vector.

[0079] The system then uses this set of weights to perform a weighted summation of the projected feature vectors to obtain the fused feature vector.

[0080] Step 4: The fused feature vector is input into the classifier, which outputs a tampering probability as high as 0.92 (the threshold is assumed to be 0.8). The system determines this as "highly suspicious" and sends it to the manual review queue. The manual reviewer ultimately confirms that it is forged. The sample and its label (negative sample) are stored in the database.

[0081] Figure 4 This is a flowchart illustrating the training and updating process of the classification module in an embodiment of this application. (In conjunction with...) Figure 4As shown, when the number of labeled real and fake images accumulates to a certain level (e.g., 1000), or when the model's accuracy on recent new samples begins to decline, the offline training process is automatically triggered. This newly collected batch of samples, containing the latest forgery techniques, will be used to retrain the entire model (including feature extraction, fusion networks, and classifiers). The new data training process is as follows: Step 1: Collect new manually verified data. Input these new samples, labeled as positive and negative, into the training process; Step 2: Extract and fuse feature information (adjusting the rules or thresholds for visual features, hidden information features (QR code hidden information features), and metadata features as needed); Step 3: Input the fused features into the classification model for training; Step 4: Obtain the optimal model data. The optimal model is generally evaluated based on the F1-score and AUC of the independent validation set. The optimal model is then deployed online after A / B testing validation.

[0082] This application is the first to organically integrate and cross-verify evidence chains from different dimensions, including image vision, hidden information QR code features, and metadata features, to construct a three-dimensional anti-counterfeiting system. The detection rate of professional-grade and covert counterfeits is significantly higher than that of traditional solutions.

[0083] In addition, this application also proposes an intelligent system with learning and iteration capabilities. Through a closed-loop iteration mechanism of "online detection-offline learning", the system can learn autonomously from errors and new forgery methods, effectively respond to new forgery technologies, and solve the problem of static model performance degradation.

[0084] Correspondingly, this application also provides a multimodal fusion verification device for business license verification, referencing Figure 5 As shown, it includes: The preliminary verification module 110 is used to extract text information from the image of the business license to be verified and to perform preliminary verification based on the text information. The first feature vector construction module 120 is used to output a first feature vector based on the image features of the business license image in response to the preliminary verification being passed. The first feature vector is used to characterize the visual features of the business license image. The second feature vector construction module 130 is used to identify the QR code in the business license image, parse it to obtain the Uniform Resource Locator (URL), perform multi-dimensional risk detection based on the URL, and construct the second feature vector based on the multi-dimensional risk detection results. The third feature vector construction module 140 is used to analyze the editing traces, creation and modification time, and geographical location when the business license image was taken based on the metadata of the business license image, and construct the third feature vector based on the analysis results. The weighted fusion module 150 is used to perform weighted fusion on the first feature vector, the second feature vector and the third feature vector to obtain a fused feature vector. Input module 160 is used to input the fused feature vector into a preset classification model to obtain the tampering probability of the business license image; Output module 170 is used to output the verification result based on the comparison result between the tampering probability and the preset threshold.

[0085] Based on at least one of the above embodiments, the electronic device in the embodiments of this application may be a user terminal device, a server, other computing devices, or a cloud server. Figure 6 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. The electronic device may include a processor 601 and a memory 602 storing computer program instructions. The processor 601 reads and executes the computer program instructions stored in the memory 602 to implement the process or function of any of the methods in the above embodiments.

[0086] Specifically, processor 601 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. Memory 602 may include a mass storage device for data or instructions. For example, memory 602 may be at least one of the following: a hard disk drive (HDD), read-only memory (ROM), random access memory (RAM), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, universal serial bus (USB) drive, or other physical / tangible memory storage device. Alternatively, memory 602 may include removable or non-removable (or fixed) media. Furthermore, memory 602 may be internal or external to the integrated gateway disaster recovery device. Memory 602 may be non-volatile solid-state memory. In other words, typically memory 602 includes a tangible (non-transitory) computer-readable storage medium (such as a memory device) encoded with computer-executable instructions, and when the software is executed (e.g., by one or more processors), it can perform the operations described in the methods of the embodiments of this application.

[0087] As an example, Figure 6The illustrated electronic device may also include a communication interface 603 and a bus 610. The processor 601, memory 602, and communication interface 603 are connected via bus 610 and communicate with each other. Bus 610 may include hardware, software, or both, and may couple components of an online data traffic metering device together. The bus may include at least one of the following: Accelerated Graphics Port (AGP) or other graphics bus, Enhanced Industry Standard Architecture (EISA) bus, Front Side Bus (FSB), HyperTransport (HT) Interconnect, Industry Standard Architecture (ISA) bus, Infinite Bandwidth Interconnect, Low Pin Count (LPC) bus, memory bus, Microchannel Architecture (MCA) bus, Peripheral Component Interconnect (PCI) bus, PCI-Express (PCI-X) bus, Serial Advanced Technology Attachment (SATA) bus, Video Electronics Standards Association Local (VLB) bus, or other suitable bus. Bus 610 may include one or more buses. Although specific buses are described or shown in the embodiments of this application, any suitable bus or interconnection method is contemplated in the embodiments of this application.

[0088] In conjunction with the methods in the above embodiments, this application also provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the process or function of any of the methods in the above embodiments.

[0089] This application also provides a computer program product that stores computer program instructions, which, when executed by a processor, implement the process or function of any of the methods described above.

[0090] The flowcharts and / or block diagrams of methods, terminals, systems, and computer program products according to embodiments of this application have been exemplarily described above, and related aspects have been described. It should be understood that each block or combination thereof in the flowcharts and / or block diagrams may be implemented by computer program instructions, by dedicated hardware performing a specified function or action, or by a combination of dedicated hardware and computer instructions. For example, these computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to form a machine such that these instructions, executed via such processor, enable the implementation of the function / action specified in each block or combination thereof in the flowcharts and / or block diagrams. Such a processor may be a general-purpose processor, a dedicated processor, a special-purpose application processor, or a field-programmable logic circuit.

[0091] The functional blocks shown in the structural block diagrams of this application can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc.; when implemented in software, they are programs or code segments used to perform the required tasks. Programs or code segments can be stored in memory or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. Code segments can be downloaded via computer networks such as the Internet or intranets.

[0092] It should be noted that this application is not limited to the specific configurations and processes described above or shown in the figures. The above descriptions are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the described systems, devices, terminals, modules, or units can be referred to the corresponding processes in the method embodiments, and need not be repeated here. It should be understood that the scope of protection of this application is not limited thereto. Any equivalent modifications or substitutions that can be conceived by those skilled in the art within the scope of the technology disclosed in this application should be covered within the scope of protection of this patent application.

Claims

1. A multimodal fusion verification method for business license verification, characterized in that, include: Extract the text information from the business license image to be verified, and perform preliminary verification based on the text information; In response to the successful preliminary verification, the following steps are performed: Based on the image features of the business license image, a first feature vector is output, which is used to characterize the visual features of the business license image. The QR code in the business license image is identified, a Uniform Resource Locator (URL) is parsed to obtain the URL, and a multidimensional risk detection is performed based on the URL. A second feature vector is then constructed based on the multidimensional risk detection results. Based on the metadata of the business license image, the editing traces, creation and modification time of the business license image, and the geographical location when the business license was taken are analyzed, and a third feature vector is constructed based on the analysis results; The first feature vector, the second feature vector, and the third feature vector are weighted and fused to obtain a fused feature vector. The fused feature vector is input into a preset classification model to obtain the tampering probability of the business license image; Based on the comparison between the tampering probability and the preset threshold, the verification result is output.

2. The method according to claim 1, characterized in that, The step of performing multi-dimensional risk detection based on the Uniform Resource Locator (URL) and constructing a second feature vector based on the multi-dimensional risk detection results includes: Parse the Uniform Resource Locator (URL) to obtain its parameter information; The consistency risk value is determined based on the degree of matching between the parameter information and the corresponding information in the text information. The reachability risk value is determined based on the response result of the network request initiated to the Uniform Resource Locator. The domain risk value is determined based on whether the domain name exists in the preset official domain name whitelist based on the Uniform Resource Locator (URL). The batch risk value is determined based on the frequency of the same Uniform Resource Locator (URL) content appearing in the verified business license images within a preset time window. The batch risk value is min(1.0, number of images with the same URL content / preset batch size threshold). The second feature vector is constructed based on the consistency risk value, the reachability risk value, the domain name risk value, and the batch risk value.

3. The method according to claim 1, characterized in that, The analysis examines the editing traces, creation and modification times of the business license image, and the geographical location when the business license was photographed. Based on the analysis results, a third feature vector is constructed, including: The metadata of the business license image is analyzed to determine whether there is signature information of image editing software, and the software editing feature value is determined based on the signature information analysis results. Analyze whether there are corresponding values ​​for the creation time and modification time in the metadata of the business license image, and determine the timeliness anomaly feature value based on the time analysis results; Extract the shooting location information from the metadata of the business license image, and extract the company registration address from the text information; Calculate the distance between the location where the business license was photographed and the company's registered address in the location information, and obtain the geographic location anomaly feature value based on the relationship between the distance and a preset distance threshold; The software editing feature value, the timeliness anomaly feature value, and the geographical location anomaly feature value are used as the feature values ​​of the third feature vector to obtain the third feature vector.

4. The method according to claim 3, characterized in that, The signature information includes the Make field, XMPToolkit field, TIFF tag, and Software field. The software editing feature value is 0 or 1. The analysis process includes: Extract the values ​​of each metadata element in the signature information according to the preset extraction order, and return the value as the result if there is a value. If the returned result contains metadata values, it is determined that the business license image has been edited by editing software, and the software editing feature value is 1.

5. The method according to claim 3, characterized in that, Calculate the distance between the location where the business license was photographed and the company's registered address in the location information, and obtain geographic location anomaly feature values ​​based on the relationship between the distance and a preset distance threshold, including: Convert the company's registered address to latitude and longitude coordinates; Calculate the spherical distance D between the latitude and longitude coordinates of the business license shooting location and the latitude and longitude coordinates of the registered address in the shooting location information; If distance D ≤ threshold T, then the geographic location anomaly feature value is 0; if distance D > threshold T, then the geographic location anomaly feature value = min(1.0, (DT) / (K×T)), where K is a constant greater than 1.

6. The method according to claim 1, characterized in that, The first feature vector, the second feature vector, and the third feature vector are weighted and fused to obtain a fused feature vector, including: The first feature vector, the second feature vector, and the third feature vector are projected to the same feature dimension through a linear transformation layer; Based on the first, second, and third feature vectors after projection, the weight values ​​of the first, second, and third feature vectors after projection are generated through a cross-attention mechanism and a decision neural network, respectively. Based on the weight values, the three projected feature vectors are weighted and summed to obtain the fused feature vector.

7. The method according to claim 1, characterized in that, The method further includes an offline classification model iteration step, which includes: The monitoring classification model iteration trigger conditions include at least one of the following: the number of newly accumulated positive and negative samples with manually labeled labels reaches a preset number; the performance index of the online classification model on the validation set is continuously lower than a preset threshold; and a preset periodic time point is reached. In response to the fulfillment of any of the aforementioned triggering conditions, a training process is initiated, the training process comprising: Obtain the labeled positive and negative sample images; Perform multimodal feature extraction and dynamic weighted fusion steps on the positive sample images and the negative sample images to construct a training dataset; The classification model is retrained using the training dataset, and the optimal model is selected based on the comprehensive performance metrics of the independent validation set. The optimal model is then deployed as a new online classification model.

8. The method according to claim 1, characterized in that, The preliminary verification includes: The key fields in the text information are compared online with the corresponding enterprise registration information obtained through official information sources; If the comparison is inconsistent, the preliminary verification is deemed to have failed, and a verification failure conclusion is directly output.

9. The method according to claim 1, characterized in that, Also includes: Based on the metadata of the business license image, a metadata confidence feature value is generated. The metadata confidence feature value is used to characterize the editing traces, creation and modification time of the business license image, and the reliability of the geographical location when the business license was taken. The first feature vector, the second feature vector, and the third feature vector are weighted and fused to obtain a fused feature vector, including: Based on the first feature vector, the second feature vector, the third feature vector, and the metadata confidence feature value, weight values ​​for the first feature vector, the second feature vector, and the third feature vector are generated respectively through a cross-attention mechanism and a decision neural network. Among the attention weights, the weight value of the third feature vector is positively correlated with the metadata confidence feature value. Based on the weight values ​​and the first feature vector, the second feature vector, and the third feature vector, a fused feature vector is constructed.

10. A multimodal fusion verification device for business license verification, characterized in that, include: The preliminary verification module is used to extract text information from the business license image to be verified and to perform preliminary verification based on the text information. The first feature vector construction module is used to output a first feature vector based on the image features of the business license image in response to the preliminary verification being passed. The first feature vector is used to characterize the visual features of the business license image. The second feature vector construction module is used to identify the QR code in the business license image, parse it to obtain a Uniform Resource Locator (URL), perform multi-dimensional risk detection based on the URL, and construct a second feature vector based on the multi-dimensional risk detection results. The third feature vector construction module is used to analyze the editing traces, creation and modification time of the business license image, and the geographical location when the business license was taken, based on the metadata of the business license image, and construct a third feature vector based on the analysis results. The weighted fusion module is used to perform weighted fusion on the first feature vector, the second feature vector and the third feature vector to obtain a fused feature vector. The input module is used to input the fused feature vector into a preset classification model to obtain the tampering probability of the business license image; The output module is used to output the verification result based on the comparison result between the tampering probability and the preset threshold.

11. An electronic device, characterized in that, The electronic device is a terminal device or a server. The electronic device includes a processor and a memory storing computer program instructions. When the electronic device executes the computer program instructions, it implements the method as described in any one of claims 1-9.

12. A computer program product, characterized in that, It includes computer program instructions that, when executed, implement the method as described in any one of claims 1-9.

13. A computer-readable storage medium, characterized in that, It stores computer program instructions that, when executed, implement the method as described in any one of claims 1-9.

Images