A Deep Learning-Based Method and System for Identifying Abnormal Financial Reimbursement in Universities
By introducing credibility mapping and cross-modal fusion mechanisms into the deep learning model, the problems of misjudgment and missed detection of abnormal reimbursements caused by imaging degradation due to mobile phone screen capture in the university financial reimbursement system are solved, and more accurate abnormal reimbursement identification and automatic review are achieved.
Patent Information
- Application Number
- CN202611323330.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-28
- Publication Date
- 2026-11-10
AI Technical Summary
Existing deep learning-based systems for identifying irregular financial reimbursements in universities cannot effectively detect the unreliability of OCR when image degradation is introduced by taking pictures of the screen with a mobile phone, leading to misjudgments of compliant reimbursements or missed detection of irregular reimbursements.
By acquiring the original image of the expense invoice, structured text and character bounding boxes are extracted, character region images are cropped and physical interference features are extracted, credibility weights are generated using a credibility mapping network, and cross-modal fusion is performed by combining a pre-trained language model and a convolutional neural network. Attention weights are dynamically modulated to perform anomaly rule reasoning and decision-making.
It effectively blocks the unattacked propagation of OCR errors to the abnormal decision-making layer, reduces the false rejection rate of compliant reimbursement, prevents the missed detection of genuine anomalies, and provides accurate visual guidance for problem areas.
Smart Images

Figure CN123069210A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing and text review technology based on deep learning, specifically to a method and system for identifying abnormal financial reimbursements in universities based on deep learning. Background Technology
[0002] In university financial reimbursement scenarios, teachers and students often take photos of invoices displayed on electronic screens and upload them via mobile phones. This results in a large number of reimbursement invoice images containing complex imaging degradation such as moiré patterns, reflections, and blurring.
[0003] Existing deep learning-based anomaly reimbursement detection systems generally employ a cascaded processing architecture of "OCR recognition → text structuring → anomaly review." The anomaly review model in this architecture is typically built upon a pre-trained language model or graph neural network, implicitly assuming the input text is accurate. However, when image degradation caused by screen capture affects key information areas of the invoice, the OCR engine outputs unreliable text. The existing architecture lacks a deep learning model component capable of perceiving and handling this front-end OCR unreliability, and cannot perform dynamic fusion and decision-making based on information quality assessment between text and image modalities. This flaw in the cascaded architecture, where OCR errors propagate unimpeded to the deep learning anomaly review model, leads to numerous compliant reimbursements being incorrectly rejected or genuine anomalies being missed, necessitating a solution at the deep learning model architecture level. Summary of the Invention
[0004] The purpose of this invention is to provide a method and system for identifying abnormal financial reimbursements in universities based on deep learning, in order to solve the problem that existing abnormal reimbursement identification models cannot perceive and make careful judgments when OCR outputs unreliable text due to non-ideal imaging such as mobile phone screen captures, resulting in OCR errors being directly misjudged as abnormal reimbursements.
[0005] To achieve the above objectives, the present invention provides the following technical solution:
[0006] The objective of this invention is achieved through the following technical solution:
[0007] A deep learning-based method for identifying abnormal expense reimbursements in universities includes:
[0008] Step S1: Obtain the original image of the expense invoice to be reviewed, as well as the structured text and bounding box coordinates of each character extracted from the original image by the OCR engine;
[0009] Step S2: For each character, crop the character region image from the original image based on the bounding box coordinates of the character, and extract statistical features from the frequency domain and spatial domain of the character region image to form a physical interference feature vector. The physical interference feature vector is used to characterize the degree of image degradation caused by non-ideal imaging in the character region.
[0010] Step S3: Input the physical interference feature vector of each character into the pre-trained credibility mapping network to generate the credibility weight corresponding to the character. The credibility weight is a continuous value, which represents the reliability of the OCR recognition result of the character.
[0011] Step S4: Encode the structured text into a text semantic vector sequence using a pre-trained language model, and encode each character region image into a visual evidence vector sequence using a convolutional neural network; use the credibility weight as a modulation factor to perform cross-modal fusion based on an attention mechanism between the text semantic vector sequence and the visual evidence vector sequence, wherein the text semantic vector sequence is used as a query, and the visual evidence vector sequence is used as a key and value, calculate the similarity between the query and the key to obtain the attention weight corresponding to each character, and use the credibility weight as a modulation factor to perform a weighted operation on the attention weight to obtain a credibility-weighted fusion representation;
[0012] Step S5: Based on the fusion representation, perform abnormal reimbursement rule reasoning to obtain the abnormal probability; aggregate the credibility weights of each character in the preset key fields to obtain the overall decision uncertainty; when the overall decision uncertainty exceeds the preset threshold, output the indication that manual review is required and the location information of the high uncertainty field; otherwise, output the abnormal probability as the final abnormal judgment result.
[0013] Preferably, step S2, which involves extracting statistical features from the frequency domain of the character region image, includes: performing a two-dimensional discrete cosine transform on the character region image to obtain a spectral coefficient matrix; calculating the high-frequency energy proportion and spectral anisotropy from the spectral coefficient matrix, and using the high-frequency energy proportion and the spectral anisotropy as frequency domain features in the physical interference feature vector; the high-frequency energy proportion is the ratio of the sum of frequency domain coefficient energies higher than a preset frequency threshold to the total energy, and the spectral anisotropy is a measure of the uniformity of spectral energy distribution in each direction; the pre-trained credibility mapping network in step S3 is a radial basis function network, which takes the high-frequency energy proportion and the spectral anisotropy as input, performs nonlinear mapping through a radial basis kernel function, and outputs the credibility weights.
[0014] Preferably, the step S4 of encoding each character region image into a visual evidence vector sequence using a convolutional neural network specifically includes: employing a visual character encoder pre-trained by contrastive learning, wherein the visual character encoder is a convolutional neural network; the contrastive learning pre-training process includes: using a clear, interference-free character image as input to the visual character encoder, using the word embeddings of the corresponding character text as positive samples, and training the visual character encoder using the InfoNCE loss function, so that the distance between the visual embedding vector of the same character and the text word embedding in the high-dimensional space is shortened, while the distance between the visual embedding vectors of different characters and the text word embeddings is widened; for each character region image, it is input into the trained visual character encoder to obtain the visual embedding vector of the corresponding character, and the visual embedding vectors of all characters constitute the visual evidence vector sequence.
[0015] Preferably, step S5 further includes: when the aggregated value of the credibility weights of all characters in a certain preset key field is lower than a preset bypass threshold, activating a visual bypass branch; the visual bypass branch takes the visual evidence vector corresponding to the character in the field obtained from the visual evidence vector sequence as input, performs sequence encoding through a lightweight Transformer encoder, and outputs the amount classification result and the decimal point position probability distribution based on the encoded features; and performs coarse-grained compliance judgment on the reimbursement invoice to be reviewed based on the amount classification result and the decimal point position probability distribution to bypass the structured text.
[0016] Preferably, step S2, which involves extracting the physical interference feature vector from the character region image, further includes: extracting noise residual blocks from the character region image using a pre-trained noise residual estimation branch, wherein the noise residual estimation branch is a convolutional network with an encoder-decoder structure; the training supervision signal for the noise residual estimation branch is provided by a pre-trained conditional generative adversarial network, wherein the generator of the conditional generative adversarial network generates an image with simulated screen-slapping noise based on a clear invoice image; the noise residual estimation branch learns to recover the noise residual map from the image with simulated screen-slapping noise, and uses the residual block corresponding to the character region in the noise residual map as the supervision signal; the noise residual block is input into a noise type encoder, which is a shallow convolutional network, and outputs a noise pattern embedding vector, wherein the noise pattern embedding vector encodes the spatial structural features of the noise; the noise pattern embedding vector is combined with statistical features extracted from the frequency domain and spatial domain of the character region image to form the physical interference feature vector.
[0017] Preferably, step S4 further includes: concatenating the noise pattern embedding vector with the text semantic vector of the corresponding character to obtain an extended text semantic vector; in the cross-modal fusion, using the extended text semantic vector as a query to interact with the visual evidence vector sequence based on an attention mechanism.
[0018] Preferably, the step S5, which involves aggregating the credibility weights of each character in the preset key field to obtain the overall decision uncertainty, specifically involves summing the credibility weights of all characters contained in the amount field or the seller's name field and taking the average value to obtain the field-level average credibility, and using the result of subtracting the field-level average credibility as the overall decision uncertainty.
[0019] Preferably, step S5, which involves aggregating the credibility weights of each character within a preset key field to obtain the overall decision uncertainty, further includes: assigning field importance weights to different preset key fields; the aggregation operation is as follows: first, the credibility weights of all characters within each preset key field are averaged to obtain a field-level uncertainty score, and then the field-level uncertainty scores of each preset key field are weighted and summed according to their respective field importance weights to obtain the overall decision uncertainty.
[0020] Preferably, the output of the indication that manual review is required and the location information of the high uncertainty field in step S5 includes: generating and displaying the original image of the expense invoice to be reviewed, and visually marking the character area with the credibility weight lower than the preset display threshold on the original image, while outputting the field-level average credibility of the field as a reference.
[0021] A deep learning-based system for identifying irregular expense reimbursements in universities includes:
[0022] The image and text acquisition module is used to acquire the original image of the expense invoice to be reviewed, as well as the structured text and bounding box coordinates of each character extracted from the original image by the OCR engine.
[0023] The interference quantization module is used to crop out a character region image from the original image based on the bounding box coordinates of the character for each character, and extract statistical features from the frequency domain and spatial domain of the character region image to form a physical interference feature vector. The physical interference feature vector is used to characterize the degree of image degradation caused by non-ideal imaging in the character region.
[0024] The credibility mapping module is used to input the physical interference feature vector of each character into a pre-trained credibility mapping network to generate the credibility weight corresponding to the character. The credibility weight is a continuous value, which represents the reliability of the OCR recognition result of the character.
[0025] The cross-modal encoding and fusion module is used to encode the structured text into a text semantic vector sequence using a pre-trained language model, and to encode each character region image into a visual evidence vector sequence using a convolutional neural network; using the credibility weight as a modulation factor, it performs cross-modal fusion based on an attention mechanism between the text semantic vector sequence and the visual evidence vector sequence, wherein the text semantic vector sequence is used as a query, and the visual evidence vector sequence is used as a key and value, the similarity between the query and the key is calculated to obtain the attention weight corresponding to each character, and the credibility weight is used as a modulation factor to perform a weighted operation on the attention weight to obtain a credibility-weighted fusion representation;
[0026] The abnormal reasoning and decision-making module is used to reason about abnormal reimbursement rules based on the fusion representation to obtain the abnormal probability; aggregate the credibility weights of each character in the preset key fields to obtain the overall decision uncertainty; when the overall decision uncertainty exceeds the preset threshold, output the indication that manual review is required and the location information of the high uncertainty field; otherwise, output the abnormal probability as the final abnormal judgment result.
[0027] The beneficial effects of this invention are as follows:
[0028] Compared with existing technologies, this invention constructs a prudent reasoning mechanism within the deep learning-based abnormal reimbursement identification model that can perceive the unreliability of the front-end OCR. By quantifying the imaging interference of each character and generating a credibility weight, this weight dynamically modulates the fusion process of text and visual modalities, effectively blocking the path of OCR errors propagating to the abnormal decision layer without attenuation. This significantly reduces the false rejection rate of compliant reimbursements caused by screen capture image interference. At the same time, through uncertainty assessment, it automatically transfers the case to manual review when the information quality is insufficient, preventing the omission of genuine anomalies and providing auditors with accurate visual guidance on problem areas. Attached Figure Description
[0029] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.
[0030] Figure 1 This is an overall flowchart of a deep learning-based method for identifying abnormal financial reimbursements in universities, according to the present invention.
[0031] Figure 2 This is a structural block diagram of a deep learning-based system for identifying abnormal financial reimbursements in universities, according to the present invention.
[0032] Figure 3 This is a detailed flowchart of the reliability modulation cross-modal fusion in this invention. Detailed Implementation
[0033] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.
[0034] See Figures 1 to 3 The technical solution of the present invention will be described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.
[0035] Example 1:
[0036] This embodiment details the implementation process of a deep learning-based method for identifying abnormal expense reimbursements in universities. The method is applied to a server deployed in the university's financial data center, which is equipped with a GPU to accelerate the inference of the deep learning model. The front-end reimbursement application client is installed on the smartphones of faculty and students, providing the function of uploading invoices by taking photos; the back-end financial review client is installed on the computers of the finance department's review personnel, providing a review and interactive interface.
[0037] This embodiment uses a typical scenario of a university teacher using a mobile phone to photograph an electronic invoice displayed on a computer screen for daily research expense reimbursement. The teacher uses the mobile phone camera to directly photograph a PDF electronic invoice displayed on a laptop screen and uploads the photo to the school's financial reimbursement system. Due to the difference between the screen refresh rate and the scanning frequency of the mobile phone camera, as well as the reflection formed by the indoor ceiling light on the screen surface, the amount area "9800.00" in the invoice photo is completely covered by a large area of colored moiré and a bright reflective spot. This imaging condition, which introduces complex image degradation such as moiré, screen reflection, local overexposure, and non-uniform blur by the user taking a picture of the screen with a mobile phone, is the non-ideal imaging condition that this solution addresses.
[0038] After the teacher uploads the photo via mobile phone, the system executes step S1. The server receives the original image of the invoice to be reviewed, which is an RGB three-channel color image. The system then calls the integrated OCR engine to process the original image. The OCR engine first performs document region detection and skew correction, then locates the text lines in the image, performs character-level segmentation on each text line, and recognizes each segmented character image, finally outputting two types of data: one is structured text, that is, the text content parsed according to the invoice layout structure, which includes at least the invoice number, invoice date, seller's name, buyer's name, amount, tax amount, product name and other key fields and their corresponding text; the other is the bounding box coordinates of each character, which defines the precise position of each recognized character in the original image, usually represented by the upper left and lower right pixel coordinates of the rectangle. In this example, due to the interference of moiré patterns, the OCR engine misinterpreted the last "0" in the amount "9800.00" as an "8" with extra strokes, causing the amount field in the structured text to be output as "9888.00".
[0039] Next, the system iterates through each character and its corresponding bounding box coordinates, executing step S2. For a given character, the system performs a slicing operation on the pixel matrix of the original image based on its bounding box coordinates, cropping out a rectangular sub-image containing only that character, i.e., the character region image. Subsequently, the system extracts multi-dimensional features from this character region image to form the physical interference feature vector described in this invention. This vector is a multi-dimensional numerical descriptor used to quantify the degree of image degradation caused by non-ideal imaging in a single character image region. Its physical meaning lies in the fact that different types of imaging interference will leave unique and quantifiable traces in the frequency distribution and spatial intensity distribution of the image.
[0040] Specifically, the extraction of frequency domain statistical features includes performing a two-dimensional discrete cosine transform on the character region image to obtain a spectral coefficient matrix with the same size as the image. The two-dimensional discrete cosine transform can convert the spatial information of an image to the frequency domain. The upper left region of the spectral coefficient matrix represents the low-frequency information of the image, while the lower right region represents the high-frequency information. Moiré patterns, due to their regular striped structure, appear as high-frequency energy concentrations in specific directions within the frequency spectrum.
[0041] Based on this spectral coefficient matrix, the system calculates the proportion of high-frequency energy. and spectral anisotropy The equation for calculating the proportion of high-frequency energy is:
[0042]
[0043] In the formula, Located on the frequency coordinate The two-dimensional discrete cosine transform coefficients at the location, where M and N are the height and width (in pixels) of the character region image, respectively. The high-frequency region is defined as follows: Where Rth is a preset frequency radius threshold, which is an empirical parameter. The setting of this threshold should take into account the size of the character region image, so that the main part of the character strokes is classified into the low-frequency region, while high-frequency interference such as moiré patterns is classified into the high-frequency region. It can typically be set to [value missing]. . The higher the value, the more the image degradation features tend to be dominated by dense, high-frequency interference.
[0044] Spectral anisotropy The calculation equation is as follows:
[0045]
[0046] In the formula, directional energy ,in and It is based on Cartesian coordinates The polar coordinates obtained by transformation For the first An angle range, This represents the total number of angle intervals, which is a preset value. For example, it can be set to 36, which means dividing 360 degrees into 36 intervals, with each interval covering 10 degrees. The larger the value, the more significant the directional interference; the smaller the value, the more uniform the energy distribution, and the more likely it is non-directional random noise.
[0047] Spatial domain statistical features include calculating the mean, standard deviation, skewness, and kurtosis of pixel intensity in the image of the character region, as well as the proportion of overexposed pixels. Finally, these frequency domain and spatial domain features are concatenated into a one-dimensional vector, which is the physical interference feature vector of the character. In this example, for the character region image of the normal character "9" in the amount field, the values in its physical interference feature vector are at a low level; while for the character region image severely interfered with by moiré patterns, its... and The value will increase significantly.
[0048] Obtain the physical interference feature vector for each character. Then, the system executes step S3. The vector is input into a pre-trained radial basis function network. This network is a feedforward neural network containing an input layer, hidden layers, and an output layer, with the hidden layers using a Gaussian kernel function as the activation function. The number of input layer nodes in this network is related to the physical disturbance feature vector. The dimensions are consistent. The hidden layer contains... 1 neuron, each neuron Having a central vector and a width parameter Total number of neurons in the hidden layer The setting is related to the diversity of interference patterns in the training data, and can usually be set between 20 and 50 to cover the cluster centers of typical interference patterns such as moiré, reflection, and blur.
[0049] When the input vector Entering the When there are 1 hidden layer neurons, the activation equation is:
[0050]
[0051] In the formula, For the first The output activation values of each hidden layer neuron and These are all parameters learned during the network pre-training phase.
[0052] The network output layer performs a linear weighted sum of the activation values, ultimately outputting a continuous value between 0 and 1, which represents the confidence weight of the character. The calculation equation is as follows:
[0053]
[0054] In the formula, For connection weights, This is a bias term. The confidence weight characterizes the reliability of the OCR recognition result based on the image quality of the character itself, without considering the contextual semantics of the character. A higher weight value indicates greater reliability. In this example, the radial basis function network outputs a confidence weight of 0.98 for the normal character "9," and a confidence weight of 0.05 for the character affected by moiré patterns and causing OCR misrecognition.
[0055] Subsequently, the system enters the core step S4, which involves the fusion of dual-stream coding and credibility modulation.
[0056] In the text stream, the system encodes the structured text obtained in step S1 into a sequence of text semantic vectors. This embodiment uses a pre-trained BERT model, which segments the structured text at the character level and feeds it into the model, outputting a hidden state vector sequence of the same length as the input sequence. Each vector in this sequence not only contains the semantic information of the character itself but also incorporates its role information within the context of the invoice text.
[0057] In the visual stream, the system encodes each character region image into a sequence of visual evidence vectors. This embodiment employs a visual character encoder pre-trained through contrastive learning; this encoder is a convolutional neural network. The pre-training process takes a clear, interference-free character image as input, the word embedding vector of the correct character text corresponding to that clear character image as a positive sample, and the word embedding vectors of other different character texts as negative samples, and trains using the InfoNCE loss function. The word embedding vector of the character text corresponding to the clear character image is obtained by querying a pre-trained Chinese word embedding table or character embedding table, which can be constructed based on the embedding layer parameters of Word2Vec, GloVe, or BERT. The expression for the InfoNCE loss function is:
[0058]
[0059] In the formula, The number of samples in the training batch. For the first Visual embedding vectors of character images, For positive sample text word embedding, The first in the batch Text word embedding, For temperature hyperparameters, The cosine similarity function is used. After training, this visual character encoder can output a visual embedding vector for any input character region image. This vector is closer to the text word embedding of the correct character it should have recognized in high-dimensional space. The visual embedding vectors of all characters constitute the visual evidence vector sequence. This visual evidence vector sequence serves as a second information channel independent of the OCR text recognition result, providing an abstract representation of the visual morphology of characters for abnormal reimbursement recognition.
[0060] Cross-modal fusion is achieved using an attention mechanism. The text semantic vector sequence is used as the query Q, and the visual evidence vector sequence is used as the key K and value V. First, the similarity score between the query and the key is calculated using a dot product operation. After SoftMax normalization, the attention weights corresponding to each character are obtained. The key innovation of this invention lies in introducing a credibility modulation step: the credibility weight of each character generated in step S3 is used as a multiplicative modulation factor and multiplied with the corresponding similarity score before normalization. The physical meaning is that if a character's credibility weight is very low (e.g., 0.05), it indicates severe interference in that region and extremely unreliable OCR results. Therefore, regardless of the original similarity between the text and the visual evidence, its final attention contribution to the visual evidence will be significantly suppressed, thus preventing the fusion model from "reading" heavily contaminated and unreliable visual information. Finally, the value V is weighted and summed using the modulated attention weights to obtain the final fused representation.
[0061] In this example, when dealing with the highly perturbative "0" in the amount field, its confidence weight of 0.05 significantly weakens the attentional interaction between its textual semantic vector (misidentified as "8" by OCR) and the visual evidence vector. Meanwhile, because the visual character encoder is pre-trained using contrastive learning, its output visual evidence vector can characterize the visual form of the character image as more like "0" than "8," thus preserving more evidence about the true visual form in the fused representation.
[0062] Finally, the system performs the prudent decision-making in step S5. An abnormal reimbursement rule inference classifier takes the fused representation as input and outputs an abnormal probability between 0 and 1, for example, 0.1. Simultaneously, the system calculates the overall decision uncertainty. The credibility weights of all characters within the amount field are aggregated, resulting in the credibility weight sequence for that amount field. The average confidence level at the field level was calculated by summing and averaging, resulting in a value of 0.788. Subtracting this value from 1 yielded an overall decision uncertainty of 0.212. With a preset threshold of 0.3, since 0.212 is less than 0.3, it indicates that the quality of the invoice information currently acquired by the system is acceptable and can support reliable machine judgment. The system directly outputs an anomaly probability of 0.1 as the final anomaly judgment result, automatically classifying the reimbursement as normal. The entire process avoids false rejections caused by interference from a single character.
[0063] Furthermore, in this embodiment, the system also includes a visual bypass branch to address scenarios where structured text becomes completely unusable under extreme interference. Specifically, before performing the aforementioned aggregation operation to calculate the overall decision uncertainty, the system first checks whether the average confidence weight of all characters within a preset key field is lower than a preset bypass threshold. This bypass threshold is typically set to a low value, such as 0.2, indicating that the OCR recognition result for that field is almost entirely unreliable. When this condition is met, the system activates the visual bypass branch.
[0064] This visual bypass branch completely bypasses structured text, using only visual evidence vectors obtained from the visual evidence vector sequence corresponding to the characters within the field as input. These visual evidence vectors are first sequence-encoded by a lightweight Transformer encoder, which has fewer layers and attention heads, making it more streamlined than a full Transformer model. This lightweight Transformer encoder captures the spatial dependencies and overall morphological features between all character visual evidence vectors within the field through a self-attention mechanism, encoding the sequence into a fixed-length global representation vector. Subsequently, two independent fully connected classification heads are connected in parallel on this global representation vector. One classification head is responsible for outputting the monetary amount classification result, such as classifying the amount into different monetary amounts like three-digit, four-digit, and five-digit numbers; the other classification head is responsible for outputting the decimal point position probability distribution, used to determine the boundary between the integer and decimal parts of the amount. Based on the monetary amount classification result and the decimal point position probability distribution, the system can perform coarse-grained compliance judgment on the invoice to be audited without relying on the specific numerical text recognized by OCR. For example, if the amount classification result shows a four-digit number and the on-campus meal reimbursement limit is 9,000 yuan, the system can determine that there is a potential over-limit and thus output a manual review instruction.
[0065] Example 2:
[0066] This embodiment is an optimization based on Embodiment 1. Regarding the physical interference feature extraction in step S2, this embodiment employs a multi-scale frequency domain analysis strategy: before calculating the frequency domain features of the character region image, an image pyramid composed of different degrees of Gaussian blur and downsampling is constructed. R_high and A_spec are calculated for each layer of the pyramid, and the results are concatenated as the final frequency domain statistical features. This approach captures interference characteristics at different levels of detail and provides a more stable representation of moiré patterns of varying thickness.
[0067] In the confidence mapping network training step S3, this embodiment introduces a refined truth value calibration method based on the confusion matrix. In the labeled data required for training the radial basis function network, for negative samples classified as "interference-confusing," the confidence truth value is set based on the actual misidentification probability of this type of interference by the OCR model on the test set. Perform calibration, that is, set as This allows the trained credibility mapping network to predict unreliability more accurately.
[0068] Regarding the selection of the visual character encoder in step S4, this embodiment uses a lightweight convolutional network pre-trained on ImageNet, such as EfficientNet or MobileNet, as the basic skeleton. The last classification layer is removed, and a fully connected layer is added, mapping the output to a space consistent with the output dimension of BERT. During pre-training, the shallow layer weights are frozen, and only the deep and newly added layers are fine-tuned, thereby utilizing general visual features to accelerate training and improve generalization ability.
[0069] Regarding the calculation of overall decision uncertainty in step S5, this embodiment further assigns importance weights to different key fields. For example, the importance weight of the amount field is set to 0.7, and the importance weight of the seller's name field is set to 0.3. During aggregation, the field-level uncertainty score of each field is calculated first, and then a weighted sum is performed according to the importance weight to obtain a more comprehensive comprehensive indicator, making the triggering condition of "requiring manual review" more scientific and reasonable.
[0070] Example 3:
[0071] This embodiment expands upon the framework of Embodiment 1 in several aspects. In the first extended scenario, the input data in step S1 is expanded to continuously captured short video clips of invoices or a series of burst photos. In step S2, for each character, the corresponding position is located in multiple frames of images using optical flow or a target tracking algorithm, and a character image sequence is cropped. The extraction of physical interference features is upgraded to spatiotemporal domain features, such as calculating the temporal variance of pixel intensity or extracting spatiotemporal frequency domain features using 3D DCT. Correspondingly, the credibility mapping network in step S3 is upgraded to a 3D convolutional neural network or long short-term memory network capable of processing spatiotemporal feature inputs, generating robust credibility weights that integrate multi-frame information.
[0072] In the second extended scenario, a "modal selection gate" mechanism is introduced in the cross-modal fusion stage of step S4. For each character, the gated network assigns confidence weights... As input, output two coefficients. and , as a dynamic mixing coefficient of text semantic vector and visual evidence vector, and satisfies .when High time, Approaching 1, primarily trusting the text; when When low, Approaching 1, it completely switches to relying on visual evidence. This hard switch and hybrid mechanism improves the certainty and interpretability of decision-making in extreme scenarios.
[0073] In the third extended scenario, this embodiment further enhances the representation capability of physical interference feature vectors. In step S2, the system not only extracts statistical features in the frequency and spatial domains, but also extracts noise residual blocks from the character region image through a pre-trained noise residual estimation branch. This noise residual estimation branch is a convolutional network with an encoder-decoder structure. Its encoder part consists of consecutive convolutional layers and max-pooling layers, progressively downsampling the input character region image to capture the contextual information of the noise; its decoder part consists of consecutive upsampling layers and convolutional layers, progressively recovering a noise residual map with the same size as the original input image. This noise residual estimation branch is specifically designed to learn how to separate pure noise components from the character image containing interference.
[0074] The training supervision signal for the noise residual estimation branch is provided by a pre-trained conditional generative adversarial network (FGAN). This FGAN consists of a generator and a discriminator. The generator uses the character region image of a clear invoice image as a condition to generate an image with simulated screen-slapping noise. The discriminator is a binary classification convolutional network used to distinguish the noisy image generated by the generator from the noisy image extracted from a real screen-slapped invoice image. After training, the generator of this FGAN is used to generate batch clear-noisy character image pairs, and the pixel-level difference between the clear and noisy images is used as the training supervision signal for the noise residual estimation branch. The noise residual estimation branch learns to recover a noise residual map consistent with the supervision signal from the noisy image.
[0075] During the inference phase, for the character region image to be processed, the trained noise residual estimation branch outputs its noise residual block. This noise residual block is then fed into a noise type encoder, which is a shallow convolutional network consisting of several convolutional layers, pooling layers, and fully connected layers. Its output is a fixed-dimensional noise pattern embedding vector. This noise pattern embedding vector encodes the spatial structural features of the noise, which include at least one or more of stripe direction, frequency distribution, and spot morphology. The noise pattern embedding vector is concatenated with statistical features extracted from the frequency and spatial domains of the character region image to form a more informative physical interference feature vector in this embodiment.
[0076] Accordingly, in step S4, the system concatenates the noise pattern embedding vector with the corresponding character's text semantic vector to obtain an extended text semantic vector. In the cross-modal fusion attention mechanism, the extended text semantic vector is used as the query Q, and the visual evidence vector sequence is used as the key K and value V for attention-based interaction. The physical significance of this operation is that the extended text semantic vector carries the type and structural information of the interference suffered by the character, enabling the query end to adjust the attention allocation strategy in a targeted manner according to the noise type when retrieving evidence from the visual end, thereby achieving differentiated trust adjustment. For example, if the noise type encoder identifies high-frequency horizontal stripe moiré patterns in the character area, after this information is encoded into the query vector, the attention mechanism can learn to reduce trust in the OCR text and rely more on visual contour information under such interference.
[0077] In the fourth extended scenario, when step S5 outputs an instruction requiring manual review, the advantages of the aforementioned noise pattern embedding vector are combined. When generating visual annotations, the system not only circles the high-uncertainty character region, but also displays interference type labels identified by the noise type encoder next to the region, such as "suspected 45-degree direction moiré interference" or "suspected local overexposure interference," providing human reviewers with richer decision-making support information and further improving review efficiency.
[0078] The detailed description of the above specific embodiments fully illustrates the feasibility, preferred implementation, and technical effects achieved by the technical solution of the present invention. Those skilled in the art can make several modifications and substitutions based on the above description without departing from the principles and spirit of the present invention, and these modifications and substitutions should also be considered within the scope of protection of the present invention.
[0079] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention. The actual method is not limited to this. In conclusion, if those skilled in the art are inspired by this description and design similar methods and embodiments without departing from the spirit of the present invention, they should all fall within the protection scope of the present invention.
Claims
1. A deep learning-based method for identifying abnormal expense reimbursements in universities, characterized in that, include: Step S1: Obtain the original image of the expense invoice to be reviewed, as well as the structured text and bounding box coordinates of each character extracted from the original image by the OCR engine; Step S2: For each character, crop the character region image from the original image based on the bounding box coordinates of the character, and extract statistical features from the frequency domain and spatial domain of the character region image to form a physical interference feature vector. The physical interference feature vector is used to characterize the degree of image degradation caused by non-ideal imaging in the character region. Step S3: Input the physical interference feature vector of each character into the pre-trained credibility mapping network to generate the credibility weight corresponding to the character. The credibility weight is a continuous value, which represents the reliability of the OCR recognition result of the character. Step S4: Encode the structured text into a text semantic vector sequence using a pre-trained language model, and encode each character region image into a visual evidence vector sequence using a convolutional neural network; use the credibility weight as a modulation factor to perform cross-modal fusion based on an attention mechanism between the text semantic vector sequence and the visual evidence vector sequence, wherein the text semantic vector sequence is used as a query, and the visual evidence vector sequence is used as a key and value, calculate the similarity between the query and the key to obtain the attention weight corresponding to each character, and use the credibility weight as a modulation factor to perform a weighted operation on the attention weight to obtain a credibility-weighted fusion representation; Step S5: Based on the fusion representation, perform abnormal reimbursement rule reasoning to obtain the abnormal probability; aggregate the credibility weights of each character in the preset key fields to obtain the overall decision uncertainty; when the overall decision uncertainty exceeds the preset threshold, output the indication that manual review is required and the location information of the high uncertainty field; otherwise, output the abnormal probability as the final abnormal judgment result.
2. The method according to claim 1, characterized in that, Step S2, which involves extracting statistical features from the frequency domain of the character region image, includes: performing a two-dimensional discrete cosine transform on the character region image to obtain a spectral coefficient matrix; calculating the high-frequency energy proportion and spectral anisotropy from the spectral coefficient matrix, and using the high-frequency energy proportion and the spectral anisotropy as frequency domain features in the physical interference feature vector; the high-frequency energy proportion is the ratio of the sum of frequency domain coefficient energies higher than a preset frequency threshold to the total energy, and the spectral anisotropy is a measure of the uniformity of spectral energy distribution in each direction; in step S3, the pre-trained credibility mapping network is a radial basis function network, which takes the high-frequency energy proportion and the spectral anisotropy as input, performs nonlinear mapping through the radial basis kernel function, and outputs the credibility weights.
3. The method according to claim 1, characterized in that, The step S4, which involves encoding each character region image into a visual evidence vector sequence using a convolutional neural network, specifically includes: employing a visual character encoder pre-trained using contrastive learning, wherein the visual character encoder is a convolutional neural network; the contrastive learning pre-training process includes: using a clear, interference-free character image as input to the visual character encoder, using the word embeddings of the corresponding character text as positive samples, and training the visual character encoder using the InfoNCE loss function, thereby narrowing the distance between the visual embedding vector of the same character and the text word embedding in the high-dimensional space, and widening the distance between the visual embedding vectors of different characters and the text word embeddings; for each character region image, inputting it into the trained visual character encoder to obtain the visual embedding vector of the corresponding character, and the visual embedding vectors of all characters forming the visual evidence vector sequence.
4. The method according to claim 3, characterized in that, Step S5 further includes: when the aggregated value of the credibility weights of all characters in a certain preset key field is lower than a preset bypass threshold, a visual bypass branch is activated; the visual bypass branch takes the visual evidence vector corresponding to the character in the field obtained from the visual evidence vector sequence as input, performs sequence encoding through a lightweight Transformer encoder, and outputs the amount classification result and the decimal point position probability distribution based on the encoded features; and performs coarse-grained compliance judgment on the reimbursement invoice to be reviewed based on the amount classification result and the decimal point position probability distribution to bypass the structured text.
5. The method according to claim 1, characterized in that, Step S2, which involves extracting the physical interference feature vector from the character region image, further includes: extracting noise residual blocks from the character region image using a pre-trained noise residual estimation branch, wherein the noise residual estimation branch is a convolutional network with an encoder-decoder structure; the training supervision signal for the noise residual estimation branch is provided by a pre-trained conditional generative adversarial network, wherein the generator of the conditional generative adversarial network generates an image with simulated screen-slapping noise based on a clear invoice image; the noise residual estimation branch learns to recover the noise residual map from the image with simulated screen-slapping noise, and uses the residual block of the corresponding character region in the noise residual map as the supervision signal; the noise residual block is input into a noise type encoder, which is a shallow convolutional network, and outputs a noise pattern embedding vector, wherein the noise pattern embedding vector encodes the spatial structural features of the noise; the noise pattern embedding vector is combined with statistical features extracted from the frequency domain and spatial domain of the character region image to form the physical interference feature vector.
6. The method according to claim 5, characterized in that, Step S4 further includes: concatenating the noise pattern embedding vector with the text semantic vector of the corresponding character to obtain an extended text semantic vector; in the cross-modal fusion, using the extended text semantic vector as a query to interact with the visual evidence vector sequence based on an attention mechanism.
7. The method according to claim 1, characterized in that, In step S5, the credibility weights of each character in the preset key field are aggregated to obtain the overall decision uncertainty. Specifically, the credibility weights of all characters contained in the amount field or the seller's name field are summed and averaged to obtain the field-level average credibility. The result of subtracting the field-level average credibility is taken as the overall decision uncertainty.
8. The method according to claim 7, characterized in that, Step S5, which involves aggregating the credibility weights of each character within a preset key field to obtain the overall decision uncertainty, further includes: assigning field importance weights to different preset key fields; the aggregation operation is as follows: first, the credibility weights of all characters within each preset key field are averaged to obtain a field-level uncertainty score, and then the field-level uncertainty scores of each preset key field are weighted and summed according to their respective field importance weights to obtain the overall decision uncertainty.
9. The method according to claim 1, characterized in that, The step S5, which outputs the indication that manual review is required and the location information of high uncertainty fields, includes: generating and displaying the original image of the expense invoice to be reviewed, and visually marking the character areas with credibility weights lower than the preset display threshold on the original image, while outputting the credibility aggregate value of the field as a reference.
10. A deep learning-based system for identifying abnormal expense reimbursements in universities, characterized in that: include: The image and text acquisition module is used to acquire the original image of the expense invoice to be reviewed, as well as the structured text and bounding box coordinates of each character extracted from the original image by the OCR engine. The interference quantization module is used to crop out a character region image from the original image based on the bounding box coordinates of the character for each character, and extract statistical features from the frequency domain and spatial domain of the character region image to form a physical interference feature vector. The physical interference feature vector is used to characterize the degree of image degradation caused by non-ideal imaging in the character region. The credibility mapping module is used to input the physical interference feature vector of each character into a pre-trained credibility mapping network to generate the credibility weight corresponding to the character. The credibility weight is a continuous value, which represents the reliability of the OCR recognition result of the character. The cross-modal encoding and fusion module is used to encode the structured text into a text semantic vector sequence using a pre-trained language model, and to encode each character region image into a visual evidence vector sequence using a convolutional neural network; using the credibility weight as a modulation factor, it performs cross-modal fusion based on an attention mechanism between the text semantic vector sequence and the visual evidence vector sequence, wherein the text semantic vector sequence is used as a query, and the visual evidence vector sequence is used as a key and value, the similarity between the query and the key is calculated to obtain the attention weight corresponding to each character, and the credibility weight is used as a modulation factor to perform a weighted operation on the attention weight to obtain a credibility-weighted fusion representation; The abnormal reasoning and decision-making module is used to reason about abnormal reimbursement rules based on the fusion representation to obtain the abnormal probability; aggregate the credibility weights of each character in the preset key fields to obtain the overall decision uncertainty; when the overall decision uncertainty exceeds the preset threshold, output the indication that manual review is required and the location information of the high uncertainty field; otherwise, output the abnormal probability as the final abnormal judgment result.
