Image recognition model training method, image recognition method, medium and product

Abstract

The invention discloses an image recognition model training method, an image recognition method, a medium and a product, relates to the technical field of artificial intelligence, in particular to the technical fields of image processing, large models, deep learning and the like, and can be applied to the field of financial science and technology. The method comprises the following steps: carrying out image preprocessing on an initial image to obtain a sample image and a sample label; pre-training the visual language joint model by adopting the public multi-modal data set to obtain a basic multi-modal model; wherein the visual language joint model comprises a visual encoder and a large language model; and based on federated learning, performing fine adjustment on the basic multi-modal model by adopting the sample image and the sample label to obtain a multi-modal image recognition model. According to the technical scheme, the identification efficiency of the multi-modal image can be improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, particularly to the fields of image processing, large models, and deep learning, and can be used in the financial technology field. Specifically, it relates to a training method, medium, and product for an image recognition model and image recognition. Background Technology

[0002] In banking business research and development and testing scenarios, employees need to process a large number of image files containing sensitive information such as business vouchers, scanned documents, and form screenshots on a daily basis. Traditional workflows rely on manual visual recognition of image content and manual input or verification of information, which has the following pain points: 1) Low efficiency: Manual information extraction is time-consuming, especially in batch image scenarios, which can easily become a bottleneck in the process (e.g., a single remittance slip requires manual verification of information such as payee, account number, and amount, which may take up to 5-10 minutes); 2) Insufficient accuracy: Manual recognition is prone to errors in information extraction due to fatigue or negligence, such as misreading key numbers such as amount and account number. For example, the system cannot effectively compare facial images, cannot completely and correctly recognize handwritten text, and easily confuse the number "7" with "1," as well as similar characters (0 and O, 1 and l, etc.), resulting in an error rate of over 15%; 3) Logical deficiencies: Existing optical character recognition (OCR) technology cannot verify business rules such as "whether the amount in words and figures matches"; 4) Fragmented multimodal processing: The system cannot simultaneously recognize multiple image types, such as the need for a dedicated seal verification system for authenticating official seals, the need to call an independent biometric platform for facial comparison, and the need for specific methods for handwritten signatures, leading to fragmented processes; 5) Security risks: Sensitive information may be exposed to unauthorized personnel due to manual operations, or spread through uncontrolled means such as screenshots and photos, posing a high risk; 6) Information silos: Image content is difficult to directly link with internal bank systems (such as core account systems and risk control systems), making it impossible to verify or call related data in real time. Therefore, an efficient image recognition method is urgently needed. Summary of the Invention

[0003] This invention provides a training method, medium, and product for image recognition models, as well as image recognition methods, media, and products, to improve the recognition accuracy of multimodal images.

[0004] According to one aspect of the present invention, a method for training an image recognition model is provided, the method comprising:

[0005] Image preprocessing is performed on the initial image to obtain sample images and sample labels;

[0006] A basic multimodal model is obtained by pre-training the visual-language joint model using a public multimodal dataset; the visual-language joint model includes a visual encoder and a large language model.

[0007] Based on federated learning, the basic multimodal model is fine-tuned using the sample images and sample labels to obtain a multimodal image recognition model.

[0008] According to another aspect of the present invention, an image recognition method is provided, the method comprising:

[0009] Obtain the original image input by the user;

[0010] The original image is preprocessed with encryption to obtain an encrypted image;

[0011] A multimodal image recognition model is used to recognize encrypted images to obtain image recognition results; wherein, the multimodal image recognition model is trained based on the model training method provided by the present invention.

[0012] The image recognition results are verified to obtain the verification results.

[0013] The verification results are anonymized based on user permissions to obtain the target result.

[0014] According to another aspect of the present invention, a training apparatus for an image recognition model is provided, the apparatus comprising:

[0015] The sample data determination module is used to preprocess the initial image to obtain sample images and sample labels;

[0016] The base model determination module is used to pre-train the visual-language joint model using a public multimodal dataset to obtain a base multimodal model; wherein, the visual-language joint model includes a visual encoder and a large language model;

[0017] The image recognition model determination module is used to fine-tune the basic multimodal model based on federated learning, using the sample images and the sample labels, to obtain a multimodal image recognition model.

[0018] According to another aspect of the present invention, an image recognition device is provided, the device comprising:

[0019] The original image acquisition module is used to acquire the original image input by the user;

[0020] An encrypted image determination module is used to perform encryption preprocessing on the original image to obtain an encrypted image;

[0021] The image recognition result determination module is used to recognize the encrypted image using a multimodal image recognition model to obtain the image recognition result; wherein, the multimodal image recognition model is trained based on the model training method provided in any embodiment of the present invention.

[0022] The verification module is used to verify the image recognition results and obtain the verification results.

[0023] The desensitization module is used to desensitize the verification results according to user permissions to obtain the target result.

[0024] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0025] At least one processor; and

[0026] A memory communicatively connected to the at least one processor; wherein,

[0027] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the training method of the image recognition model or the image recognition method according to any embodiment of the present invention.

[0028] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the training method of the image recognition model or the image recognition method according to any embodiment of the present invention.

[0029] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements a training method for an image recognition model or an image recognition method according to any embodiment of the present invention.

[0030] The technical solution of this invention involves preprocessing an initial image to obtain sample images and sample labels; pre-training a visual-language joint model using a public multimodal dataset to obtain a basic multimodal model; wherein the visual-language joint model includes a visual encoder and a large language model; and fine-tuning the basic multimodal model using sample images and sample labels based on federated learning to obtain a multimodal image recognition model. Through this technical solution, multimodal image recognition is achieved using a large language model and a visual encoder, enabling the recognition of various images and improving image recognition accuracy.

[0031] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a flowchart of a training method for an image recognition model according to an embodiment of the present invention;

[0034] Figure 2 This is a flowchart of a training method for an image recognition model according to an embodiment of the present invention;

[0035] Figure 3 This is a flowchart of an image recognition method provided according to an embodiment of the present invention;

[0036] Figure 4 This is a schematic diagram of the structure of a training device for an image recognition model according to an embodiment of the present invention;

[0037] Figure 5 This is a schematic diagram of the structure of an image recognition device according to an embodiment of the present invention;

[0038] Figure 6 This is a schematic diagram of the structure of an electronic device that implements the training method or image recognition method of the image recognition model in the embodiments of the present invention. Detailed Implementation

[0039] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0040] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0041] Furthermore, it should be noted that the information collected in the technical solution of this invention is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data all comply with the relevant laws, regulations and standards of relevant countries and regions, necessary confidentiality measures have been taken, and public order and good morals are not violated. Corresponding operation entry points are provided for users to choose to authorize or refuse.

[0042] While existing general-purpose OCR tools can extract text from images, they lack adaptability to financial scenarios, cannot distinguish between sensitive and non-sensitive information, and lack contextual semantic understanding capabilities. Furthermore, they are not deeply integrated with internal bank business systems, making it difficult to meet security and compliance requirements. General-purpose OCR tools (such as Tesseract) lack adaptability to financial scenarios and struggle to uniformly process multiple image types within the same platform (simultaneously recognizing handwritten images, facial images, signature images, easily confused characters, etc., within the same image). They also cannot handle input data with complex tables, handwritten text, and overlapping seals. For example, the rule engine + template matching approach is only suitable for fixed-format images, resulting in low flexibility and high maintenance costs. Existing general-purpose multimodal model recognition methods are not trained and fine-tuned for specific internal bank financial data, failing to meet the deployment requirements of the bank's intranet.

[0043] In addition, traditional NLP techniques (such as the traditional OCR tools mentioned above) can only handle relatively single modalities (text or image), while Large Language Models (LLMs) naturally support cross-modal joint modeling, and can simultaneously understand visual information in images (such as the location of an official seal) and contextual semantics (such as transfer instructions in an email body), achieving "image-text mutual verification." Traditional NLP relies on a large number of customized rules (such as regular expressions), making it difficult to handle unknown formats or non-standardized inputs.

[0044] Dynamic business rule injection: LLM supports encoding banking business rules into model inputs through prompt engineering, making the output results inherently compliant with compliance requirements. Traditional NLP, on the other hand, requires the additional development of post-processing modules, increasing system complexity.

[0045] Figure 1 This is a flowchart illustrating a training method for an image recognition model according to an embodiment of the present invention. This embodiment is applicable to situations in financial scenarios where various business images need to be recognized, such as images containing sensitive information like business vouchers, scanned documents, and form screenshots. This method can be executed by an image recognition model training device, which can be implemented in hardware and / or software. This image recognition model training device can be configured in an electronic device that carries the training function of the image recognition model, such as a server. Figure 1 As shown, the method includes:

[0046] S110. Perform image preprocessing on the initial image to obtain sample images and sample labels.

[0047] In this embodiment, the initial image refers to unprocessed historical business images of financial institutions, including but not limited to handwritten images, facial images, signature images, and images of easily confused characters. The sample image is an image that has undergone image preprocessing; the sample label is the label corresponding to the sample image.

[0048] An optional approach involves preprocessing the initial image to obtain sample images and sample labels, including: standardizing the initial image to obtain a standardized image; enhancing the standardized image to obtain an enhanced image; labeling the enhanced image to obtain sample labels; and desensitizing the enhanced image to obtain sample images.

[0049] A normalized image is an image obtained by normalizing an initial image. An enhanced image is an image obtained by enhancing a normalized image.

[0050] Specifically, the initial image undergoes standardization processing, such as denoising, perspective correction, and resolution unification, to obtain a standardized image. Then, the representative image is enhanced through operations like light correction and cropping to obtain an enhanced image. Next, key regions, such as text, faces, or signatures, are manually or semi-automatically identified, and structured labels, such as name, account number, amount, and seal type, are associated with these regions to obtain sample labels. Finally, the sample image is desensitized using a rule engine that masks sensitive fields or preserves formatting encryption, resulting in the final sample image.

[0051] Understandably, by standardizing, enhancing, and desensitizing the images to obtain sample images, sensitive data in the images are protected, and the sample images are enriched to provide a guarantee for subsequent model training.

[0052] For example, enhancing a standardized image to obtain an enhanced image includes: enhancing the standardized image based on a generative adversarial network to generate an adversarial image; physically rendering the adversarial image to obtain a rendered image; and processing the rendered image based on a financial lexicon to obtain the enhanced image.

[0053] Adversarial images refer to images generated by processing standardized images using adversarial networks. Rendered images refer to images that have had a degradation effect added to standardized images.

[0054] Specifically, adversarial networks are used to simulate degradation scenarios such as blurring, tilting, and low resolution for standardized images, generating adversarial images. Then, a physical rendering engine is used to apply degradation effects such as creases, light and shadow interference, and JPEG compression artifacts to the adversarial images to obtain the rendered image. This can improve the model's resilience and realize the simulation of image degradation scenarios. Afterwards, based on a financial lexicon, the text in the rendered image is replaced. For example, the letter "l" is replaced with a thin "1", and a circular line is added around the number "0" to obtain an enhanced image, forcing the model to learn to distinguish subtle differences.

[0055] It is understandable that image enhancement processing, such as adding noise, can improve model robustness and performance.

[0056] It should be noted that these rigorously labeled sample images not only provide a high-quality training set for subsequent model training, but also effectively evaluate the robustness of the model under different lighting, angles and noise environments during the validation phase.

[0057] S120. The visual-language joint model is pre-trained using a public multimodal dataset to obtain the basic multimodal model.

[0058] Public multimodal datasets refer to publicly available sets of multimodal images used for multimodal image recognition. The visual-language joint model includes a visual encoder and a large language model, where the visual encoder can be a Transform encoder. The base multimodal model refers to the model obtained by training the temporal-language joint model on a public multimodal dataset.

[0059] Specifically, keeping the training framework and model parameters of the visual-language joint model unchanged can avoid catastrophic forgetting. Pre-training the visual-language joint model using a public multimodal dataset yields a basic multimodal recognition capability. This enables the final multimodal image recognition model to possess general image understanding and text generation abilities.

[0060] S130. Based on federated learning, the basic multimodal model is fine-tuned using sample images and sample labels to obtain a multimodal image recognition model.

[0061] In this embodiment, the multimodal image recognition model refers to the final generated model suitable for multimodal image recognition in financial scenarios.

[0062] Specifically, based on federated learning, each branch financial institution, such as each sub-bank, trains a basic multimodal model locally using local sample images and labels. For example, it can perform lightweight LoRA fine-tuning, updating parameters by 0.5% to obtain parameter gradients. These gradients are then uploaded to a central server. The central server aggregates the parameter gradients uploaded by each branch financial institution and feeds back the updated gradients to each institution, enabling iterative training until a training stopping condition is met, resulting in a multimodal image recognition model. The training stopping condition is that the training loss stabilizes within a set range or the number of iterations meets a set requirement. It should be noted that the set range and the set number of iterations can be set by those skilled in the art based on actual conditions.

[0063] The technical solution of this invention involves preprocessing an initial image to obtain sample images and sample labels; pre-training a visual-language joint model using a public multimodal dataset to obtain a basic multimodal model; wherein the visual-language joint model includes a visual encoder and a large language model; and fine-tuning the basic multimodal model using sample images and sample labels based on federated learning to obtain a multimodal image recognition model. Through this technical solution, multimodal image recognition is achieved using a large language model and a visual encoder, enabling the recognition of various images and improving image recognition accuracy.

[0064] Figure 2 This is a flowchart of a training method for an image recognition model according to an embodiment of the present invention. Based on the above embodiments, this embodiment further optimizes the process of "fine-tuning the basic multimodal model using sample images and sample labels to obtain a multimodal image recognition model," providing an optional implementation scheme. For example... Figure 2 As shown, the method includes:

[0065] S210. Perform image preprocessing on the initial image to obtain sample images and sample labels.

[0066] S220. A common multimodal dataset is used to pre-train the visual-language joint model to obtain a basic multimodal model.

[0067] The visual-language joint model includes a visual encoder and a large language model.

[0068] S230. Based on federated learning, the basic multimodal model is fine-tuned using sample images and sample labels to obtain a multimodal image recognition model.

[0069] An alternative approach involves fine-tuning a basic multimodal model using sample images and sample labels to obtain a multimodal image recognition model. This includes: extracting features from sample images using a visual encoder to obtain image features; aligning the image features with business text descriptions to obtain image-text pairs; and using a large language model to perform multi-task learning on the image-text pairs and sample labels to obtain the multimodal image recognition model.

[0070] The visual encoder is used to extract image features from sample images. Image features refer to the image obtained after feature extraction from the sample image, which can be represented in matrix or vector form. An image-text pair refers to an image-text pair obtained by aligning image features with business text descriptions.

[0071] Specifically, the sample image is input into a visual encoder for feature extraction to obtain image features. Then, the corresponding business text description for the sample image is searched from a business text description library. The image features and the corresponding business text description are concatenated to obtain an image-text pair. Next, based on a prompt word template, the sample text pair and sample labels are injected with rules to obtain the target prompt word. The target prompt word is input into a large language model to obtain at least one prediction result. Based on at least one prediction result and the sample label, the training loss is determined. Iterative training is performed based on the training loss until the training stopping condition is met, resulting in a multimodal image recognition model.

[0072] Understandably, combining visual and large language models for multi-task learning can improve image recognition accuracy in various scenarios.

[0073] It should be noted that the prompt word template can be updated periodically. Specifically, this can involve parsing new business specification documents from financial institutions, updating the knowledge graph, and adjusting the prompt word template based on the knowledge graph.

[0074] An optional approach involves using a large language model to perform multi-task learning on image-text pairs and sample labels to obtain a multimodal image recognition model. This includes: using the large language model to perform sub-task learning on image-text pairs and sample labels to obtain at least two sub-task recognition results; wherein the sub-task learning includes at least two of the following: character recognition, signature judgment, facial recognition, and signature verification; determining a sub-training loss based on the sub-task recognition results and sample labels; jointly combining the at least two sub-training losses to obtain a target training loss; and training the basic multimodal model using the target training loss to obtain the multimodal image recognition model.

[0075] Specifically, a large language model is used to learn sub-tasks from image-text pairs and sample labels to obtain at least two sub-task recognition results. Based on the loss function corresponding to the sub-task recognition results, a sub-training loss is calculated according to the sub-task recognition results and the corresponding sub-labels in the sample labels. The at least two sub-training losses are combined, for example, by weighted summation, to obtain the target training loss. The target training loss is used to iteratively train the basic multimodal model to obtain a multimodal image recognition model.

[0076] Understandably, multi-task recognition can enhance cross-modal consistency, enabling the system to extract transaction elements and generate structured outputs when processing mixed scenarios (such as receipts containing both signatures and handwritten text), ensuring that the model dynamically adapts to the iteration of banking business.

[0077] The technical solution of this invention involves preprocessing an initial image to obtain sample images and sample labels; pre-training a visual-language joint model using a public multimodal dataset to obtain a basic multimodal model; wherein the visual-language joint model includes a visual encoder and a large language model; and fine-tuning the basic multimodal model using sample images and sample labels based on federated learning to obtain a multimodal image recognition model. Through this technical solution, multimodal image recognition is achieved using a large language model and a visual encoder, enabling the recognition of various images and improving image recognition accuracy.

[0078] Figure 3 This is a flowchart of an image recognition method according to an embodiment of the present invention. This embodiment is applicable to situations in financial scenarios where various business images are recognized, such as images containing sensitive information like business vouchers, scanned documents, and form screenshots. The method can be executed by an image recognition device, which can be implemented in hardware and / or software. The image recognition device can be configured in an electronic device that carries image recognition functionality, such as a server.

[0079] like Figure 3 As shown, the method includes:

[0080] S310. Obtain the original image input by the user.

[0081] In this embodiment, the original image refers to the image input by the user that needs to be recognized.

[0082] Specifically, users can input the original image on their mobile devices.

[0083] S320. Perform encryption preprocessing on the original image to obtain an encrypted image.

[0084] Specifically, the original image undergoes encryption preprocessing, such as binarization or edge detection, to obtain an encrypted image.

[0085] S330. A multimodal image recognition model is used to recognize the encrypted image to obtain the image recognition result.

[0086] The multimodal image recognition model is trained based on the image recognition model training method provided in any embodiment of the present invention.

[0087] Specifically, for simple tasks, such as printed text recognition, a lightweight OCR model is used to perform image recognition on encrypted images to obtain the recognition results. For complex tasks, such as handwritten text association analysis, a multimodal image recognition model is used to recognize encrypted images to obtain the recognition results.

[0088] S340. Verify the image recognition results to obtain the verification results.

[0089] Specifically, the image recognition results are verified a second time based on the rule engine. For example, the financial information in the image recognition results is compared with the system balance to obtain the verification result.

[0090] If the verification fails, the abnormal fields in the image are highlighted, or a manual review is triggered. If the verification passes, S350 is executed.

[0091] S350. Desensitize the verification results according to user permissions to obtain the target result.

[0092] Specifically, the validation results are anonymized according to user permissions, such as masking sensitive fields. For example, if the user is a regular employee, the last four digits of the account number are hidden to obtain the target result.

[0093] The technical solution provided in this invention involves: acquiring the original image input by the user; performing encryption preprocessing on the original image to obtain an encrypted image; using a multimodal image recognition model to recognize the encrypted image to obtain an image recognition result; verifying the image recognition result to obtain a verification result; and desensitizing the verification result according to user permissions to obtain the target result. Through the above technical solution, image recognition accuracy can be improved by using a multimodal image recognition model for image recognition in complex scenes.

[0094] Based on the above embodiments, the multimodal image recognition model can also obtain new business rule documents through continuous optimization; and the multimodal image recognition model can be continuously optimized according to the target results and the new business rule documents.

[0095] Specifically, the multimodal image recognition model can be continuously optimized based on user feedback samples, namely the original images and target results, as well as new business rule documents, for example, by launching online fine-tuning every month.

[0096] Understandably, continuous optimization of multimodal image recognition models can improve model performance and ensure image recognition accuracy.

[0097] Based on the above embodiments, as an optional aspect of the present invention, it further includes: visualizing the target results to generate performance reports and risk warning information.

[0098] Specifically, based on indicators such as special character recognition rate, official seal matching accuracy, and compliance audit report, the target results are visualized to generate real-time performance reports and risk warning information.

[0099] Among them, risk warning information refers to information that does not meet the indicators in the target results.

[0100] Understandably, visualizing the target results makes it easier to understand the identification results intuitively and to monitor risks.

[0101] The technical solution of this invention, through multimodal large-scale model joint training, improves the character recognition accuracy in complex scenarios to 99.3% (22% higher than traditional OCR), and supports contextual semantic reasoning (such as distinguishing between "transfer amount" and "reference number"). The processing time for a single remittance slip is reduced from an average of 3 minutes to 5 seconds, labor costs are reduced by 90%, and the error rate drops to below 0.1%. It supports real-time anti-money laundering alerts (such as identifying "split transactions" and "high-frequency small-amount transfers"), improving risk control response speed. Automated processing improves efficiency by more than 80% compared to manual processing, with a key field recognition accuracy of 99.5% (tested using historical bank data), significantly reducing operational risks and solving the "illusion problem" of general LLM in professional scenarios. Furthermore, through data anonymization, hierarchical access control, and operational auditing, it prevents the leakage of sensitive information and strengthens security and compliance.

[0102] The technical solution of this invention can seamlessly integrate with business processes, deeply integrate with industry APIs, and support one-stop operation from information extraction to business execution, shortening the business processing cycle. Moreover, the floating window interactive design reduces interface jumps, and the drag-and-drop function adapts to users' existing operating habits, making the learning cost close to zero; thus improving the user experience.

[0103] The modular architecture of this invention supports rapid integration with new business types (such as letter of credit recognition and digital RMB wallet screenshot parsing). The platform interface has pre-built extension capabilities for connecting to blockchain and RPA systems. The model can be continuously optimized for scenario adaptability through incremental training. This platform enables the intelligent and secure utilization of unstructured image information in banking business scenarios, providing an innovative solution for the digital transformation of the financial industry. It also provides an SDK for third-party systems to call, facilitating multi-department collaboration within the banking ecosystem (such as the legal department automatically extracting key contract clauses).

[0104] Furthermore, traditional NLP technologies (such as the traditional OCR tools mentioned above) can only handle relatively single modalities (text or image), while LLM in this invention naturally supports cross-modal joint modeling, and can simultaneously understand visual information in images (such as the location of an official seal) and contextual semantics (such as transfer instructions in the email body), achieving "image-text mutual verification". For example, LLM can extract the "amount" field from a remittance slip image and combine it with the "payment purpose" in the email body to infer the transaction risk level.

[0105] The technical solution of this invention can improve the generalization ability in complex scenarios: LLM has a powerful few-shot learning capability through massive pre-trained data, and can quickly identify easily confused characters (such as 0 / O) and blurry images (such as low-resolution handwritten signatures) in banking scenarios, significantly reducing the cost of manual annotation. In contrast, traditional NLP relies on a large number of customized rules (such as regular expressions), which are difficult to handle unknown formats or non-standardized inputs.

[0106] This invention supports dynamic business rule injection: LLM supports encoding banking business rules into model input through prompt engineering, making the output results naturally compliant with compliance requirements. Traditional NLP, on the other hand, requires the additional development of post-processing modules, increasing system complexity.

[0107] The technical solution of this invention implements end-to-end encryption, hierarchical access control, and sensitive data anonymization throughout the entire process of information extraction, transmission, and storage, ensuring compliance with financial industry security standards and constructing a multimodal intelligent processing platform that is only available to authorized personnel, thus enhancing security.

[0108] Figure 4 This is a schematic diagram of a training device for an image recognition model according to an embodiment of the present invention. This embodiment is applicable to situations in financial scenarios where various business images need to be recognized, such as images containing sensitive information like business vouchers, scanned documents, and form screenshots. The training device for this image recognition model can be implemented in hardware and / or software, and can be configured in an electronic device that carries the training function of the image recognition model, such as a server. Figure 4 As shown, the device includes:

[0109] The sample data determination module 410 is used to perform image preprocessing on the initial image to obtain sample images and sample labels;

[0110] The base model determination module 420 is used to pre-train the visual-language joint model using a public multimodal dataset to obtain a base multimodal model; wherein, the visual-language joint model includes a visual encoder and a large language model;

[0111] The image recognition model determination module 430 is used to fine-tune the basic multimodal model based on federated learning, using sample images and sample labels, to obtain a multimodal image recognition model.

[0112] The technical solution of this invention involves preprocessing an initial image to obtain sample images and sample labels; pre-training a visual-language joint model using a public multimodal dataset to obtain a basic multimodal model; wherein the visual-language joint model includes a visual encoder and a large language model; and fine-tuning the basic multimodal model using sample images and sample labels based on federated learning to obtain a multimodal image recognition model. Through this technical solution, multimodal image recognition is achieved using a large language model and a visual encoder, enabling the recognition of various images and improving image recognition accuracy.

[0113] Optionally, the sample data determination module 410 is used for:

[0114] The initial image is standardized to obtain a standardized image;

[0115] Enhancement processing is performed on the standardized image to obtain the enhanced image;

[0116] The enhanced images are labeled to obtain sample labels;

[0117] The enhanced image is desensitized to obtain the sample image.

[0118] Optionally, the sample data determination module 410 is specifically used for:

[0119] Generative adversarial networks are used to enhance standardized images to generate adversarial images.

[0120] Physical rendering is performed on the adversarial image to obtain the rendered image;

[0121] The rendered image is processed based on a financial thesaurus to obtain an enhanced image.

[0122] Optionally, the image recognition model determination module 430 is used for:

[0123] A visual encoder is used to extract features from the sample images to obtain image features;

[0124] Align the image features with the business text description to obtain image-text pairs;

[0125] A multi-task learning method is used to learn image-text pairs and sample labels using a large language model, resulting in a multimodal image recognition model.

[0126] Optionally, the image recognition model determination module 430 is specifically used for:

[0127] A large language model is used to learn sub-tasks from image-text pairs and sample labels to obtain recognition results for at least two sub-tasks; the sub-tasks include at least two of the following: character recognition, signature judgment, facial recognition, and signature verification.

[0128] Based on the sub-task recognition results and sample labels, determine the sub-training loss;

[0129] The target training loss is obtained by combining at least two sub-training losses.

[0130] The basic multimodal model is trained using the target training loss to obtain a multimodal image recognition model.

[0131] The image recognition model training device provided in the embodiments of the present invention can execute the image recognition model training method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0132] Figure 5 This is a schematic diagram of an image recognition device according to an embodiment of the present invention. This embodiment is applicable to situations in financial scenarios where various business images need to be recognized, such as images containing sensitive information like business vouchers, scanned documents, and form screenshots. The image recognition device can be implemented in hardware and / or software and can be configured in an electronic device that carries image recognition functionality, such as a server. Figure 5 As shown, the device includes:

[0133] The original image acquisition module 510 is used to acquire the original image input by the user;

[0134] The encrypted image determination module 520 is used to perform encryption preprocessing on the original image to obtain an encrypted image;

[0135] The image recognition result determination module 530 is used to recognize the encrypted image using a multimodal image recognition model to obtain the image recognition result; wherein, the multimodal image recognition model is trained based on the model training method of any of the above embodiments.

[0136] The verification module 540 is used to verify the image recognition results and obtain the verification results.

[0137] The desensitization module 550 is used to desensitize the verification results according to user permissions to obtain the target result.

[0138] The technical solution provided in this invention involves: acquiring the original image input by the user; performing encryption preprocessing on the original image to obtain an encrypted image; using a multimodal image recognition model to recognize the encrypted image to obtain an image recognition result; verifying the image recognition result to obtain a verification result; and desensitizing the verification result according to user permissions to obtain the target result. Through the above technical solution, image recognition accuracy can be improved by using a multimodal image recognition model for image recognition in complex scenes.

[0139] Optionally, the device also includes a model optimization module for:

[0140] Obtain the new business rules document;

[0141] Based on the target results and new business rules documents, the multimodal image recognition model is continuously optimized.

[0142] Optionally, the device also includes a visualization module for:

[0143] Visualize the target results to generate performance reports and risk warning information.

[0144] The image recognition device provided in the embodiments of the present invention can execute the image recognition method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0145] According to embodiments of the present invention, the present invention also provides an electronic device, a readable storage medium, and a computer program product.

[0146] Figure 6 This is a schematic diagram of the structure of an electronic device that implements the training method or image recognition method of the image recognition model in the embodiments of the present invention. Figure 6 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0147] like Figure 6As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0148] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0149] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as training methods for image recognition models, or image recognition methods.

[0150] In some embodiments, the image recognition model training method, or the image recognition method, may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the image recognition model training method, or the image recognition method described above, may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the image recognition model training method, or the image recognition method, by any other suitable means (e.g., by means of firmware).

[0151] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0152] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0153] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0154] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0155] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0156] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0157] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0158] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A training method for an image recognition model, characterized in that, include: Image preprocessing is performed on the initial image to obtain sample images and sample labels; A basic multimodal model is obtained by pre-training the visual-language joint model using a public multimodal dataset; the visual-language joint model includes a visual encoder and a large language model. Based on federated learning, the basic multimodal model is fine-tuned using the sample images and sample labels to obtain a multimodal image recognition model.

2. The method according to claim 1, characterized in that, Image preprocessing is performed on the initial image to obtain sample images and sample labels, including: The initial image is standardized to obtain a standardized image; The standardized image is enhanced to obtain an enhanced image; The enhanced image is labeled to obtain sample labels; The enhanced image is then desensitized to obtain a sample image.

3. The method according to claim 2, characterized in that, The normalized image is enhanced to obtain an enhanced image, including: The standardized image is enhanced using a generative adversarial network to generate an adversarial image. The adversarial image is physically rendered to obtain a rendered image; The rendered image is processed based on a financial thesaurus to obtain an enhanced image.

4. The method according to claim 1, characterized in that, The basic multimodal model is fine-tuned using the sample images and sample labels to obtain a multimodal image recognition model, including: The sample images are used to extract features using a visual encoder to obtain image features; Align the image features and business text descriptions to obtain image-text pairs; A multi-task learning method is used to learn the image-text pairs and the sample labels using a large language model, resulting in a multimodal image recognition model.

5. The method according to claim 4, characterized in that, A multi-task learning process is performed on image-text pairs and sample labels using a large language model to obtain a multimodal image recognition model, including: A large language model is used to learn sub-tasks from the image-text pairs and the sample labels to obtain at least two sub-task recognition results; wherein the sub-task learning includes at least two of the following: character recognition, signature judgment, facial recognition, and signature verification. Based on the sub-task identification results and the sample labels, determine the sub-training loss; The target training loss is obtained by combining at least two sub-training losses. The basic multimodal model is trained using the target training loss to obtain a multimodal image recognition model.

6. An image recognition method, characterized in that, include: Obtain the original image input by the user; The original image is preprocessed with encryption to obtain an encrypted image; A multimodal image recognition model is used to recognize encrypted images to obtain image recognition results; wherein the multimodal image recognition model is trained based on the image recognition model training method described in any one of claims 1-5. The image recognition results are verified to obtain the verification results. The verification results are anonymized based on user permissions to obtain the target result.

7. The method according to claim 6, characterized in that, Also includes: Obtain the new business rules document; Based on the target results and the new business rules document, the multimodal image recognition model is continuously optimized.

8. The method according to claim 6, characterized in that, Also includes: The target results are visualized to generate performance reports and risk warning information.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute and implement the training method of the image recognition model according to any one of claims 1-5, or the image recognition method according to claims 6-8.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the training method of the image recognition model according to any one of claims 1-5, or the image recognition method according to claims 6-8.

Images