Bank card information processing method, electronic device, storage medium and program product

Abstract

The embodiment of the invention provides a bank card information processing method, electronic equipment, a storage medium and a program product. The method comprises the following steps: receiving image information of a to-be-identified bank card input by a user; according to the image information, extracting image features and text features through a pre-trained multi-modal large model, and according to the image features and the text features, obtaining card surface identification information; if the card surface identification information passes the verification of the validity verification rule, obtaining the opening bank information of the to-be-identified bank card by adopting a pre-trained large language model according to the card surface identification information; and sending the card surface identification information and the opening bank information corresponding to the to-be-identified bank card to the human-computer interaction interface, and performing visual display through the human-computer interaction interface. Through cooperation of image information multi-modal extraction, multi-modal fusion, rule verification and semantic processing, automation of the whole bank card recognition process is achieved, and the recognition efficiency and accuracy under complex scenes such as cross-bank and cross-card-type scenes are improved.

Description

Technical Field

[0001] This application relates to the field of computers, and more particularly to a bank card information processing method, electronic device, storage medium, and program product. Background Technology

[0002] When processing claims, payment is made based on the customer's bank card information (including card number and issuing bank). Currently, obtaining bank card information mainly relies on manual entry or traditional optical character recognition (OCR). Manual entry involves claims personnel filling in the relevant fields, but this process is cumbersome, inefficient, and prone to errors in the absence of a unified verification mechanism. Traditional OCR solutions use photographed or scanned images as input, obtaining fields such as card numbers through text detection and character recognition. Under standardized image conditions, this can reduce the amount of manual input.

[0003] However, bank card images captured at claims processing sites are often affected by factors such as shooting tilt, reflections, obstructions, and dirt; at the same time, different banks and card types have differences in layout design, font arrangement, and character grouping. This leads to a decrease in recognition accuracy of traditional OCR models for processing bank card information in cross-bank and cross-card type scenarios. Summary of the Invention

[0004] This application provides a bank card information processing method, electronic device, storage medium, and program product to improve the accuracy of bank card information identification.

[0005] In a first aspect, embodiments of this application provide a bank card information processing method, including:

[0006] Receive image information of the bank card to be recognized, input by the user;

[0007] Based on the image information, image features and text features are extracted from the image information using a pre-trained multimodal large model, and card recognition information corresponding to the bank card to be identified is obtained based on the image features and the text features.

[0008] Based on the card identification information, a preset validity verification rule is used for verification;

[0009] If the card recognition information passes the validity verification rule, then based on the card recognition information, a pre-trained large language model is used to obtain the opening bank information of the bank card to be recognized.

[0010] The card identification information and the opening bank information corresponding to the bank card to be identified are sent to the human-computer interaction interface and displayed visually through the human-computer interaction interface.

[0011] In one possible implementation, the multimodal large model includes an image encoder and a text encoder;

[0012] Accordingly, based on the image information, image features and text features are extracted from the image information using a pre-trained multimodal large model, including:

[0013] Based on the image information, the image features in the image information are extracted by the image encoder;

[0014] Based on the image information, the text features in the image information are extracted by the text encoder.

[0015] In one possible implementation, the multimodal large model further includes a multimodal fusion module and an output layer;

[0016] Accordingly, based on the image features and the text features, card recognition information corresponding to the bank card to be identified is obtained, including:

[0017] The multimodal fusion module fuses the image features and the text features to obtain fused features.

[0018] Based on the fusion features, the card identification information corresponding to the bank card to be identified is output through the output layer.

[0019] In one possible implementation, the output layer employs a Transformer decoder; a preset optimizer is used to update the model parameters of the Transformer decoder, and a preset learning rate decay strategy is used to schedule the learning rate of the Transformer decoder.

[0020] In one possible implementation, the card identification information is used to perform a verification using preset validity rules, including:

[0021] Based on the bank card number in the card recognition information, a preset Rune algorithm verification rule is used to verify whether the bank card number passes the validity verification rule.

[0022] In one possible implementation, the step of obtaining the opening bank information of the bank card to be identified using a pre-trained large language model based on the card recognition information includes:

[0023] Based on the card recognition information, model input information is constructed according to a preset format;

[0024] Based on the input information of the model, the opening bank information of the bank card to be identified is obtained through a large language model based on a transformer structure.

[0025] The bank information includes the full Chinese name of the bank and its interbank code.

[0026] In one possible implementation, training the large language model includes:

[0027] Training samples are constructed based on a multi-turn dialogue format, wherein the training samples include system setting information, bank card code information and corresponding bank information;

[0028] Based on the training samples, a supervised fine-tuning strategy is adopted to fine-tune the full parameters of the large language model. The model parameters of the large language model are updated through the cross-entropy loss function to obtain the trained large language model.

[0029] Secondly, embodiments of this application provide a bank card information processing device, comprising:

[0030] The information receiving module is used to receive image information of the bank card to be recognized input by the user;

[0031] The first processing module is used to extract image features and text features from the image information using a pre-trained multimodal large model, and to obtain card recognition information corresponding to the bank card to be identified based on the image features and the text features.

[0032] The information verification module is used to verify the card surface identification information using preset validity verification rules.

[0033] The second processing module is used to obtain the opening bank information of the bank card to be identified by using a pre-trained large language model based on the card recognition information if the card recognition information passes the validity verification rule.

[0034] The information display module is used to send the card identification information and the opening bank information corresponding to the bank card to be identified to the human-computer interaction interface, and to display them visually through the human-computer interaction interface.

[0035] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0036] The memory stores computer-executed instructions;

[0037] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0038] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0039] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0040] This application provides a bank card information processing method, electronic device, storage medium, and program product. It receives image information of a bank card to be recognized from user input, extracts image and text features using a pre-trained multimodal large-scale model, and fuses and analyzes these features to obtain card recognition information. This improves upon the traditional OCR model's strong dependence on fixed formats and poor robustness to complex card faces, enhancing the stability and accuracy of image recognition across different banks and card types. The recognition results are validated using validity rules, eliminating misrecognition issues caused by image interference and printing blur, thus improving the reliability of the recognition results and the accuracy of subsequent processing. After the recognition results pass validation, a pre-trained large-scale language model is used to obtain the account opening bank information based on the card recognition information. Leveraging the advantages of the large-scale language model in semantic mapping and pattern mining, high-accuracy automatic recognition of the account opening bank is achieved. Finally, the recognized card information and account opening bank information are sent to a human-computer interaction interface for visualization, enhancing the intuitiveness and interactivity of the information feedback and facilitating quick user verification and subsequent business processing. Overall, this application achieves full automation of the bank card recognition process through the synergy of multimodal extraction of image information, multimodal fusion, rule verification and semantic processing, thereby improving recognition efficiency and accuracy in complex scenarios such as cross-bank and cross-card types. Attached Figure Description

[0041] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0042] Figure 1 A schematic diagram illustrating the application scenario of the bank card information processing method provided in this application;

[0043] Figure 2 A flowchart illustrating the bank card information processing method provided in this application;

[0044] Figure 3 This is a schematic diagram of the method for training a large language model provided in an embodiment of this application;

[0045] Figure 4A schematic diagram of the bank card information processing device provided in this application;

[0046] Figure 5 A schematic diagram of the structure of the electronic device provided in this application.

[0047] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0048] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0049] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0050] Figure 1 This is a schematic diagram illustrating an application scenario for the bank card information processing method provided in this application. For example... Figure 1 As shown, the specific application scenario of this application includes: terminal device 101 and computer device 102. Terminal device 101 is used to complete user interaction and data collection, specifically including collecting image information of a bank card through a camera or scanning module, and guiding the user to complete operations such as taking pictures, confirming, and submitting through a human-computer interaction interface. Terminal device 101 can be a mobile terminal, a counter workstation, or a PC client, capable of performing local standardization processing on the collected image information (such as unifying size and encoding format), and then transmitting the image information to computer device 102 through a communication network.

[0051] Computer device 102 is used to perform computational tasks, deploying a pre-trained multimodal large model, validity verification rules, and a pre-trained large language model used in this application. After receiving image information from terminal device 101, computer device 102 sequentially performs the following steps: extracting image and text features based on the multimodal large model and obtaining card recognition information; verifying the card recognition information according to preset validity verification rules; and, if the verification passes, calling the large language model to obtain the bank account information based on the card recognition information. After processing, computer device 102 returns the card recognition information and bank account information to terminal device 101.

[0052] After receiving the processing results from the computer device 102, the terminal device 101 displays them visually on the human-computer interaction interface, including card identification information such as the bank card number, bank name, card type, and expiration date, as well as the full Chinese name and interbank code of the opening bank information, for verification, confirmation, and subsequent use by business personnel or users. In actual deployment, the computer device 102 can be a centralized server, a private cloud node, or an edge computing device; the two ends communicate through a controlled network channel, supporting transmission encryption and access control policies to ensure the security and integrity of image information and recognition results during transmission and processing. Through the collaboration of the terminal device 101 and the computer device 102, the bank card information processing method of this application can operate stably in multiple terminals and scenarios.

[0053] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0054] Figure 2 A flowchart illustrating the bank card information processing method provided in this application is shown below. Figure 2 As shown, the method includes:

[0055] S21, receive the image information of the bank card to be identified input by the user.

[0056] In this embodiment, image information is typically acquired through image acquisition devices. For example, a user can take a picture of the bank card using a terminal's camera, scanner, or other hardware module with image acquisition capabilities. During image acquisition, auxiliary interface prompts can be used to guide the user to place the bank card in a suitable position and angle, thereby ensuring the clarity and integrity of the image content. After image input is complete, the image data can be parsed and its storage structure constructed, converting the acquired bank card image into an input matrix format that can be directly read by a pre-trained multimodal large-scale model. This method enables the processing of image input from various devices (such as mobile terminals, counter workstations, scanning terminals, etc.), achieving multi-scenario compatibility and cross-device consistency, providing a reliable data foundation for subsequent multimodal feature extraction and bank card information recognition.

[0057] S12. Based on the image information, the image features and text features in the image information are extracted through a pre-trained multimodal large model, and the card recognition information corresponding to the bank card to be recognized is obtained based on the image features and text features.

[0058] In this embodiment, a pre-trained multimodal large model is used to parse the image content. First, the received image information undergoes standardized preprocessing, including size adjustment, color normalization, and noise suppression, to ensure the input data meets the model's requirements. A multimodal large model refers to a model system capable of simultaneously processing inputs from different modalities (such as images and text). It uses parallel image encoding and text encoding channels to comprehensively extract and semantically model different modal information from the bank card image. For example, the image encoding module, based on a deep neural network structure, extracts features from the overall layout of the bank card, border features, card number area position, font outline, and pattern elements, generating a high-dimensional image feature representation. The text encoding module identifies and encodes the text information contained in the image, converting character content, arrangement patterns, and font styles into semantic feature vectors.

[0059] After feature extraction, image and text features are jointly processed using a multimodal fusion mechanism. For example, the fusion process can employ an attention-based cross-modal alignment algorithm or a feature-weighted fusion algorithm, enabling the model to understand the overall semantic structure of the bank card within the same feature space, thereby identifying key regions related to bank card information. By structurally decoding the fused features, card face recognition information is generated, including key fields such as bank card number, bank name, card type, cardholder name, and expiration date. In summary, this embodiment achieves deep semantic analysis of bank card image content through multimodal feature extraction and fusion, maintaining the stability and accuracy of recognition results under complex conditions such as uneven lighting, shooting angle deviations, or card face reflections. Compared to traditional single OCR recognition methods, this application exhibits stronger adaptability and recognition generalization in complex scenarios involving multiple banks and card types, providing reliable structured data input for subsequent validity verification and acquisition of account opening bank information.

[0060] S13, Based on the card recognition information, perform verification using preset validity verification rules.

[0061] In this embodiment, to ensure the accuracy and authenticity of the recognition results, a validity check is performed on the card recognition information to eliminate erroneous or invalid data, thereby improving the accuracy and robustness of subsequent bank identification. Card recognition information typically includes the bank card number, issuing bank logo, and card type name. Pre-defined validity check rules include format-based rules and structural rationality checks. Specifically, format-based checks may include whether the bank card number length is within a reasonable range and whether it consists entirely of numbers; structural checks may verify whether the card number prefix belongs to a known bank numbering system. By setting a validity check mechanism after card recognition, the recognition results are filtered and corrected. The validity check improves the overall recognition process's fault tolerance and accuracy, effectively enhancing stability and reliability in complex application scenarios involving multiple banks and card types.

[0062] S14. If the card recognition information passes the validity verification rule, then based on the card recognition information, a pre-trained large language model is used to obtain the opening bank information of the bank card to be recognized.

[0063] In this embodiment, if the identified card information meets the preset verification rules, the bank account opening information is then identified. A pre-trained large language model is used to perform semantic analysis and encoding recognition on the verified card information, thereby automatically obtaining the corresponding bank account opening information. The large language model has the ability to perform feature modeling and information extraction on both structured and unstructured input data, overcoming the limitations of traditional rule-based systems in fuzzy matching, contextual understanding, and cross-institutional compatibility. Specifically, the pre-trained large language model is based on a Transformer Architecture, possessing multi-layered attention mechanisms and deep semantic modeling capabilities, enabling it to extract the semantic mapping relationship between bank card encoding information and its bank account opening from historical learning data. During the analysis, the model comprehensively considers factors such as the card number prefix (i.e., BIN code), number length, and contextual semantics, outputting the predicted bank account opening information. The bank account opening information includes not only the full Chinese name of the bank but may also further include structured identification fields such as the interbank code. This embodiment uses a large language model to predict the opening bank information based on card recognition information, thereby achieving automated and semantic extraction of the opening bank information and improving the accuracy of the information collection process and the efficiency of business processing.

[0064] S15 sends the card identification information and account opening bank information corresponding to the bank card to be identified to the human-computer interaction interface, and displays them visually through the human-computer interaction interface.

[0065] In this embodiment, to enable users to intuitively obtain the recognition results, the recognized information is sent to the human-computer interaction interface for unified display. Specifically, the card recognition information includes, but is not limited to, the bank card number, cardholder's name, card type, expiration date, and issuing institution name; the bank account information includes the full Chinese name and interbank code of the bank corresponding to the card. The recognition results are organized and transmitted through a preset interface communication mechanism to ensure the consistency and integrity of the data structure throughout the interaction process.

[0066] Furthermore, the human-computer interaction interface can be deployed on mobile terminals (such as smartphones and tablets) or business processing clients on desktop systems, possessing basic graphical display capabilities and supporting formatted output of structured fields. The card recognition information and bank account information sent to the interface are styled, for example, by grouping card number fields (e.g., 4 digits per group), highlighting name fields, and using bank icons to supplement bank names, to enhance the user's reading experience. During the visualization process, status labels can be added to each field, such as "Recognition Successful," "Pending Confirmation," and "Verification Failed," based on recognition confidence information or verification status, so that users can quickly determine whether manual intervention is needed. In addition, the human-computer interaction interface can also support interactive behaviors such as information confirmation, field editing, and re-recognition, further improving the flexibility and controllability of actual business processes. In summary, by pushing card recognition information and bank account information to the human-computer interaction interface and displaying them visually, the transparency and interactivity of the recognition process are enhanced. This allows claims personnel or users to verify the recognition results in an intuitive interface, reducing business errors caused by recognition mistakes and improving the accuracy, usability, and user experience of the overall information collection process.

[0067] In one embodiment, the multimodal large model includes an image encoder and a text encoder.

[0068] Next, an implementation method is provided for extracting image features and text features from the image information in step S12 above. Based on the above embodiment, it includes:

[0069] S1211, Based on the image information, the image features in the image information are extracted by the image encoder;

[0070] S1212: Based on the image information, the text features in the image information are extracted by a text encoder.

[0071] In this embodiment, the image encoder and text encoder establish representations for different modal inputs, respectively, and provide aligned feature representations for subsequent multimodal fusion and output layers. The image encoder receives image information, normalizes it by size and color space, and inputs it at a fixed resolution or with short-side alignment, outputting image features with spatial dimensions. These image features can preserve information such as the layout structure, texture, and region contrast of the bank card surface. The text encoder receives text-related input, which can be text region features obtained by localization in the image information, or a character sequence representation formed after preliminary character recognition in the image information. The text encoder encodes characters in the embedding layer and establishes contextual relationships by combining positional representations, outputting text features that reflect number grouping patterns, date formats, and Chinese lexical features.

[0072] In practical implementation, image encoders can employ convolutional neural network (CNN) structures, vision transformer (ViT) structures, or hybrid structures of both. They extract local texture and edge information at shallow layers and aggregate global layout and long-range dependencies at deeper layers, forming image feature tensors with multi-scale and hierarchical semantics. To enhance stability in tilted, reflective, and slightly occluded scenes, deformation-insensitive convolutional units or global self-attention modules can be introduced during the encoding process to improve the ability to distinguish card number arrangement, bank logos, and expiration date locations. By hierarchically encoding image information, the goal of compressing redundant pixel data while preserving spatial structure is achieved, providing a high signal-to-noise ratio visual representation for subsequent feature fusion.

[0073] The text encoder is used to extract textual information features from images. Its input can be image segments corresponding to detected text regions in the image. In its implementation, the text encoder can employ a transformer-based sequence modeling structure, capturing long-distance dependencies between characters through a multi-head self-attention mechanism, and performing contextual encoding on character sequences or text region sequences derived from image information. To accommodate the mixed distribution of numeric sequences of bank card numbers and Chinese bank names, the text encoder can introduce a unified encoding scheme for numeric characters, Chinese characters, and special delimiters during the embedding stage, and adapt the grouping rhythm of card numbers and the "MM / YY" pattern of dates in the position representation. By performing serialization modeling on text-related inputs, simultaneous representation of format patterns and semantic cues is achieved, improving the discriminability of subsequent field-level readings.

[0074] Furthermore, to ensure smooth alignment of the image encoder and text encoder outputs in subsequent stages, both feature paths employ a unified feature dimension and a mappable temporal / spatial index. Image features can be mapped to a sequence format compatible with the text stride through region pooling or feature downsampling, while text features retain the index information corresponding to character positions for cross-modal alignment. In summary, through the collaborative work of the two encoders, structured extraction of image and text features from bank card image information is achieved, providing a semantically rich and structurally clear feature foundation for subsequent processes and improving the overall recognition accuracy.

[0075] In one embodiment, the multimodal large model further includes a multimodal fusion module and an output layer.

[0076] Accordingly, an implementation method is provided for obtaining the card recognition information corresponding to the bank card to be identified in step S12 above. Based on the above embodiment, it includes:

[0077] S1221, through the multimodal fusion module, image features and text features are fused to obtain fused features;

[0078] S1222, based on the fusion features, outputs the card recognition information corresponding to the bank card to be identified through the output layer.

[0079] In this embodiment, the multimodal large model also includes a multimodal fusion module and an output layer to achieve deep fusion and structured output of features from different modalities. The multimodal fusion module maps heterogeneous modal data (i.e., image features and text features) to a unified representation space, thereby constructing a cross-modal joint representation. This module typically uses attention mechanisms, self-attention fusion networks, or cross-modal alignment algorithms to weightedly integrate the input image and text features. The fused feature representation is called the fused feature, which has stronger contextual understanding and structural representation capabilities, accurately reflecting the global and local information structure of the bank card image and improving subsequent recognition accuracy. After obtaining the fused feature, it is input to the output layer.

[0080] The output layer receives the fused features and maps them to a recognition result with a specific structure. This output layer can be implemented using a Transformer decoder, sequence labeling network, or pointer network, and its tasks include generating or classifying various components of the recognition result, such as bank card number, bank name, card type, cardholder name, and expiration date. The output layer is capable of natural language generation and structured information extraction, while also performing sequence dependency modeling and end-to-end optimization training to ensure that multimodal information is accurately mapped to complete and usable card recognition information. In summary, by setting up a multimodal fusion module and an output layer, the model's robustness and accuracy in recognizing complex backgrounds, occlusions, and overlapping images are improved, enhancing its adaptability and expressive power in recognizing complex bank card images.

[0081] In one specific embodiment, the output layer employs a Transformer decoder. The Transformer decoder is built upon an attention mechanism, possessing sequence modeling and context-aware capabilities, enabling accurate extraction of heterogeneous information from multiple fields. In practical deployments, the Transformer decoder can adopt a standard multi-layer structure, with each layer containing a multi-head attention sublayer and a feedforward neural network sublayer, connected by residual connections between layers.

[0082] During the training phase, a pre-defined optimizer is configured for the Transformer decoder to update the model parameters. Preferably, the optimizer is AdamW (Adaptive Moment Estimator with Weight Decay), which combines L2 regularization with a first-order gradient adaptive strategy, improving convergence speed and stability while maintaining the model's generalization ability. During training, to reduce model oscillations caused by excessively high learning rates or slow convergence caused by excessively low learning rates, a pre-defined learning rate decay strategy is used to dynamically adjust the learning rate. Specifically, the learning rate decay strategy employed is cosine annealing, which periodically reduces the learning rate during the training cycle, allowing the model to maintain a large learning rate in the early stages and converge smoothly in the later stages. This scheduling mechanism helps the model escape local optima and improves the final recognition accuracy. Through the synergistic effect of the optimizer and the learning rate strategy, the performance of the recognition model can be improved while maintaining training stability, ensuring that the Transformer decoder has good generalization ability under different data distribution conditions.

[0083] In one embodiment, step S13 is further described below. Based on the above embodiment, verification is performed using preset validity verification rules according to the card recognition information, including:

[0084] S131, Based on the bank card number in the card recognition information, a preset Rune algorithm verification rule is used to verify whether the bank card number passes the validity verification rule.

[0085] In this embodiment, the Luhn Algorithm is a simple and effective modulo-10 checksum algorithm, commonly used for error detection in identification codes such as card numbers and account numbers. It can be used to determine whether a number sequence conforms to basic numbering standards. Specifically, the bank card number is first extracted from the card identification information, and then processed digit by digit from right to left. According to the Luhn Algorithm rules, every other digit in the original sequence is doubled. If the result of doubling is a two-digit number, it is split and summed. The processed result is then summed with the digits in the undoubled positions, and the sum is checked to see if it is divisible by 10. If it is divisible, the bank card number is considered to have passed the validity check; otherwise, it is considered to have failed the validity check. By applying the Luhn Algorithm to the bank card number, incorrect bank card numbers caused by image recognition errors and user operation mistakes are effectively eliminated, achieving the first round of validity filtering of the identification results and improving the overall recognition accuracy.

[0086] In one embodiment, the acquisition of the account opening bank information of the bank card to be identified in step S14 above will be further explained. Based on the above embodiment, it includes:

[0087] S141, Based on card recognition information, construct model input information according to a preset format;

[0088] S142, Based on the model input information, obtain the opening bank information of the bank card to be identified through a large language model based on the transformer structure; wherein, the opening bank information includes the full Chinese name of the opening bank and the interbank code.

[0089] In this embodiment, to further determine the bank card's issuing bank information, the extracted card identification information needs to be input into the large language model. To ensure the model can accurately understand the input and correctly generate the target result, the card identification information needs to be structured to construct model input information conforming to a preset format. This preset format is designed based on the training corpus format and context requirements of the large language model, typically using a textual prompt method, and may include input identifiers, card number context, separators, and model recognition keywords. For example, when constructing the actual model input information, the bank card number in the card identification information can be concatenated into the format "Please determine the issuing bank of the following bank card number: XXXX XXXX XXXX XXXX" to form a natural language input sequence with a clear task instruction. This input format can be adjusted according to the training method of the model used to ensure that the input is compatible with the model's semantic understanding structure.

[0090] After construction, the model input information is fed into the pre-trained large language model. The large language model is a large-scale parametric model built on a transformer-based architecture, possessing contextual understanding and natural language generation capabilities. Upon receiving the input, the model automatically invokes its encoding and decoding structure, combining the semantic mapping relationship between bank card numbers and issuing banks learned during training, to generate the issuing bank information corresponding to the input bank card number. This issuing bank information includes the full Chinese name of the issuing bank and its interbank code. The full Chinese name of the issuing bank is used for human-computer interaction and manual confirmation; the interbank code is a unique bank identifier used for automatic inter-system transfer and clearing. Generating these two fields not only improves the readability and accuracy of the recognition results but also provides a basis for subsequent business operations.

[0091] Figure 3 This is a schematic diagram of the method flow for training a large language model provided in an embodiment of this application. Figure 3 As shown, it includes:

[0092] S31, construct training samples according to the multi-turn dialogue format, wherein the training samples include system setting information, bank card code information and its corresponding opening bank information;

[0093] S32, based on the training samples, adopts a supervised fine-tuning full-parameter fine-tuning strategy, updates the model parameters of the large language model through the cross-entropy loss function, and obtains the trained large language model.

[0094] In this embodiment, training samples with a business context are first constructed. Specifically, the training samples are organized in a multi-turn dialogue format, simulating a natural language interaction process between a user and the system for the bank card account opening information recognition task. This allows the large language model to learn the task objective in a semantic environment that approximates the business scenario. The training samples include: system setting information, bank card encoding information, and the corresponding bank opening information. The system setting information defines the model's role and task scope, for example, specifying that "the large language model is an assistant for recognizing bank card account opening information," guiding the model to understand instructions. The bank card encoding information consists of structured or semi-structured content related to the bank account, such as the bank card number, issuing bank identifier, and card type code, extracted from the card recognition information, serving as the model's input features. The bank opening information is the target field expected to be output by the model, including the full Chinese name and interbank code of the bank.

[0095] During the training phase, Supervised Fine-tuning (SFT) is employed for model learning. This method involves inputting pairs of training samples into the model and providing supervision based on pre-defined labels, allowing the model to progressively adjust its internal parameters to better fit the sample distribution. During fine-tuning, a full-parameter fine-tuning strategy is used, updating the gradients of all trainable parameters in the large language model to ensure the model can maximize the absorption of semantic mapping features from the training corpus, thereby improving its generalization ability and task adaptability. To effectively optimize model parameters, the Cross-Entropy Loss function is chosen as the training objective function. This loss function measures the probability difference between the model's predictions and the actual labels, driving the model to converge towards accurately generating bank account opening information. Through this optimization strategy, the model parameters are continuously iteratively updated, ultimately resulting in a fully trained large language model capable of accurately outputting the corresponding bank account opening information based on the card information. In summary, this embodiment enhances the large language model's ability to understand bank card structural information by constructing training samples with dialogue context and using a fully parameter-supervised fine-tuning strategy, thereby improving its accuracy and adaptability in the bank account identification task.

[0096] In one specific embodiment, based on industry-specific coding rules for bank card numbers, bank card numbers and their corresponding issuing bank information are extracted from successful bank transfer data in historical transactions. Key information elements in the bank card coding information are then parsed according to rules, including the issuing bank's institutional identification code, custom bit segments, and related card type and institution level indication bits. This process performs structured coding on the parsed elements, making them directly usable as training input for a large language model to learn the mapping relationship between bank card coding information and issuing bank information (full Chinese name and interbank code). By constructing training materials using successful samples, verifiable supervision signals are provided for subsequent learning.

[0097] Next, the collected bank card numbers, bank account opening information, and interbank codes underwent data cleaning and consistency checks. The cleaning process standardized the number formats, removing separators and non-numeric characters, unifying full-width and half-width characters, and verifying length, prefixes, and check digits according to rules. The corresponding full Chinese names of the bank account opening and interbank codes were standardized and formatted using a standard library, eliminating missing, duplicate, and obviously contradictory entries. For abnormal samples that the automated process could not identify, they were manually reviewed for final labeling. The cleaned and labeled samples were stratified by time and source, and divided into training, validation, and test sets for parameter learning, hyperparameter and threshold adjustment, and generalization ability evaluation, respectively. This systematic cleaning and labeling strategy ensured the usability and traceability of training samples, guaranteeing stable input for subsequent model learning stages.

[0098] In constructing the training dataset, the cleaned bank card coding information and its corresponding issuing bank information were organized into samples in a multi-turn dialogue format to closely resemble the habits and context-dependent characteristics of large language models. Each sample contains at least two fields: "input_messages" and "label". "Input_messages" represents the multi-turn dialogue organized according to system settings, user messages, and assistant messages. System settings define the task and output format, user messages describe the bank card coding information (such as the issuing bank's identification code, number length and grouping pattern, and custom bit segments), and assistant messages provide examples of the target output format. "Label" represents the issuing bank information corresponding to the coding information, including the full Chinese name of the issuing bank and its interbank code. This data organization method allows the model to obtain clear task boundaries and output templates during the training phase.

[0099] In the implementation of large language model training and analysis, a large language model based on a transformer structure is used as the base model, and a supervised fine-tuning strategy with full parameter tuning is employed for training. During training, samples in multi-turn dialogue formats are used as cue-response pairs as input to the model. The model parameters are progressively updated using the cross-entropy loss function as the optimization objective, enabling the model to learn the mapping relationship between bank card code information and account opening bank information. To ensure convergence stability and generalization ability, an optimization scheme with weight decay capability and a learning rate scheduling strategy (e.g., a combination of AdamW as the optimizer and cosine annealing as the learning rate decay strategy) are adopted during training, combined with an appropriate initial learning rate and batch size for iterative training. For example, in this embodiment, the model size can be on the order of approximately 0.6B parameters, the initial learning rate is set to 1e-5, and a cosine decay strategy is used to dynamically adjust the learning rate parameter during model training; the number of training samples used in each iteration is set to 256. After training, the weights and configuration files (such as PyTorch's .pth model file) are saved for subsequent use, and the model version identifier is recorded for subsequent gray-scale testing and rollback.

[0100] During deployment and integration, the trained multimodal and large language models are deployed to GPU servers to improve the response speed and throughput of model inference. For example, a single pre-built GPU card can support 200 concurrent inference requests, with a single image inference time of less than 30 seconds. To achieve unified access and service calls to the multimodal and large language models, a vLLM model inference service is used to encapsulate these models into standardized API interfaces. These standardized service interfaces are then used to interface with the human-computer interaction interface and the claims processing system. This interface supports concurrent calls, multi-user request management, and asynchronous processing, ensuring the model maintains stability and low latency even under high concurrency scenarios.

[0101] Figure 4 This is a schematic diagram of the structure of the bank card information processing device provided in this application, as shown below. Figure 4 As shown, the bank card information processing device 40 provided in this embodiment includes:

[0102] The information receiving module 401 is used to receive image information of the bank card to be identified input by the user;

[0103] The first processing module 402 is used to extract image features and text features from the image information using a pre-trained multimodal large model, and to obtain the card recognition information corresponding to the bank card to be recognized based on the image features and text features.

[0104] The information verification module 403 is used to verify the card based on the card identification information using preset validity verification rules;

[0105] The second processing module 404 is used to obtain the opening bank information of the bank card to be identified by using a pre-trained large language model based on the card recognition information if the card recognition information passes the validity verification rule.

[0106] The information display module 405 is used to send the card recognition information and account opening bank information of the bank card to be identified to the human-computer interaction interface, and to display them visually through the human-computer interaction interface.

[0107] In one possible implementation, the multimodal large model includes an image encoder and a text encoder;

[0108] The first processing module 402 is specifically used for: extracting image features from the image information using an image encoder based on the image information; and extracting text features from the image information using a text encoder based on the image information.

[0109] In one possible implementation, the multimodal large model also includes a multimodal fusion module and an output layer;

[0110] The first processing module 402 is further specifically used for: fusing image features and text features through a multimodal fusion module to obtain fused features; and outputting card recognition information corresponding to the bank card to be recognized through an output layer based on the fused features.

[0111] In one possible implementation, the output layer uses a Transformer decoder. When training the Transformer decoder, a preset optimizer is used to update the model parameters of the Transformer decoder, and a preset learning rate decay strategy is used to schedule the learning rate of the Transformer decoder.

[0112] In one possible implementation, the information verification module 403 is specifically used to: verify the bank card number in the card recognition information using a preset Rune algorithm verification rule, so as to determine whether the bank card number passes the validity verification rule.

[0113] In one possible implementation, the second processing module 404 is specifically used for: constructing model input information based on card recognition information and according to a preset format; and obtaining the opening bank information of the bank card to be identified through a large language model based on a transformer structure based on the model input information; wherein the opening bank information includes the full Chinese name of the opening bank and the interbank code.

[0114] In one possible implementation, a data training module is also included, which is used to construct training samples according to the multi-turn dialogue format. The training samples include system setting information, bank card code information and corresponding bank information. Based on the training samples, a supervised fine-tuning full-parameter fine-tuning strategy is adopted to update the model parameters of the large language model through the cross-entropy loss function to obtain the trained large language model.

[0115] The bank card information processing device 40 provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0116] Figure 5 A schematic diagram of the structure of the electronic device provided in this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the electronic device 50 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0117] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0118] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0119] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0120] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0121] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0122] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0123] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0124] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0125] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0126] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for processing bank card information, characterized in that, include: Receive image information of the bank card to be recognized, input by the user; Based on the image information, image features and text features are extracted from the image information using a pre-trained multimodal large model, and card recognition information corresponding to the bank card to be identified is obtained based on the image features and the text features. Based on the card identification information, a preset validity verification rule is used for verification; If the card recognition information passes the validity verification rule, then based on the card recognition information, a pre-trained large language model is used to obtain the opening bank information of the bank card to be recognized. The card identification information and the opening bank information corresponding to the bank card to be identified are sent to the human-computer interaction interface and displayed visually through the human-computer interaction interface.

2. The method according to claim 1, characterized in that, The multimodal large model includes an image encoder and a text encoder; Accordingly, based on the image information, image features and text features are extracted from the image information using a pre-trained multimodal large model, including: Based on the image information, the image features in the image information are extracted by the image encoder; Based on the image information, the text features in the image information are extracted by the text encoder.

3. The method according to claim 2, characterized in that, The multimodal large model also includes a multimodal fusion module and an output layer; Accordingly, based on the image features and the text features, card recognition information corresponding to the bank card to be identified is obtained, including: The multimodal fusion module fuses the image features and the text features to obtain fused features. Based on the fusion features, the card identification information corresponding to the bank card to be identified is output through the output layer.

4. The method according to claim 3, characterized in that, The output layer uses a Transformer decoder; a preset optimizer is used to update the model parameters of the Transformer decoder, and a preset learning rate decay strategy is used to schedule the learning rate of the Transformer decoder.

5. The method according to claim 1, characterized in that, Based on the card recognition information, a preset validity verification rule is used for verification, including: Based on the bank card number in the card recognition information, a preset Rune algorithm verification rule is used to verify whether the bank card number passes the validity verification rule.

6. The method according to claim 1, characterized in that, The step of obtaining the opening bank information of the bank card to be identified using a pre-trained large language model based on the card recognition information includes: Based on the card recognition information, model input information is constructed according to a preset format; Based on the input information of the model, the opening bank information of the bank card to be identified is obtained through a large language model based on a transformer structure. The bank information includes the full Chinese name of the bank and its interbank code.

7. The method according to claim 1, characterized in that, Training the large language model includes: Training samples are constructed based on a multi-turn dialogue format, wherein the training samples include system setting information, bank card code information and corresponding bank information; Based on the training samples, a supervised fine-tuning strategy is adopted to fine-tune the full parameters of the large language model. The model parameters of the large language model are updated through the cross-entropy loss function to obtain the trained large language model.

8. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images