Structured extraction method of insurance customer service image text information based on multimodal large model

Abstract

The present invention discloses a method for extracting structured text information from insurance customer service images based on a multimodal large model. S1: a preliminary OCR large model is obtained by training text corpus and font samples using a multimodal training strategy; S2: a general domain image data set is constructed; S3: a business domain image data set is constructed; S4: the general domain image data set and the business domain image data set are integrated, and further training is performed based on the pre-trained OCR large model to generate an insurance customer service multimodal large model optimized for insurance customer service scenarios; S5: the fine-tuned insurance customer service multimodal large model is deployed to a GPU server. The present invention significantly improves the efficiency, accuracy and adaptability of structured extraction of image text information in insurance customer service scenarios through an end-to-end multimodal large model architecture, a dynamic data training strategy and a real-time optimization mechanism.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal large models, and in particular to a method for extracting structured text information from insurance customer service images based on a multimodal large model. Background Art

[0002] In the insurance customer service scenario, extracting key text information from images and structuring it is an important technical challenge. Currently, most existing technical solutions use a pipeline approach to complete this task, which includes the following main steps: first, identify the image type, then use OCR technology to identify the characters in the image based on the image type; finally, formulate special structured logic for each type of image based on the image category and character recognition results. Although basic functions can be achieved in simple scenarios, there are many limitations in actual applications:

[0003] First, there are significant problems with recognition accuracy. Errors in the image classification model may cause subsequent OCR recognition to deviate completely from expectations, rendering the results unusable. OCR character recognition also has many errors. In payment screenshot scenarios, the separators in the amount are easily misrecognized, which in turn affects the extraction and processing of the amount.

[0004] Second, the development of structured logic is complex and difficult to expand. When there are multiple amounts or table elements in the image, the extraction rules need to be highly customized, which further increases the difficulty of structuring and significantly reduces the overall recognition accuracy. In addition, each type of image requires independent design and maintenance of structured logic, which is a huge workload and cannot be effectively expanded to new image types or scenarios.

[0005] Third, the system construction is complex. The pipeline solution relies on the collaborative work of multiple models, including image type recognition model, OCR text area detection model, text direction correction model, character segmentation model and character recognition model. The diversity and interdependence of the models make the cost of system construction and maintenance high. At the same time, any abnormality in any link in the process will make the final result unreliable.

[0006] Fourth, the overall efficiency is low. Since multiple independent models need to be trained, the pipeline solution takes a long time and cannot meet the real-time needs of quickly processing a large number of images in insurance customer service scenarios.

[0007] In summary, the existing technology has significant deficiencies in recognition accuracy, system complexity, scalability and efficiency, and cannot meet the requirements for efficient processing of key information extraction and structuring of text in images in insurance customer service scenarios. Summary of the invention

[0008] One purpose of the present invention is to propose a method for structured extraction of image text information in insurance customer service scenarios based on a multimodal large model. The present invention significantly improves the efficiency, accuracy and adaptability of structured extraction of image text information in insurance customer service scenarios through an end-to-end multimodal large model architecture, a dynamic data training strategy and a real-time optimization mechanism.

[0009] According to an embodiment of the present invention, a method for extracting structured text information from insurance customer service pictures based on a multimodal large model includes the following steps:

[0010] S1. Collect text corpus and font samples, and use multimodal training strategy to train the text corpus and font samples to obtain a preliminary OCR large model;

[0011] S2. Collect real natural images from multiple sources, use the OCR large model to recognize the text content in real natural images, and complete the construction of a general domain image dataset;

[0012] S3. Collect business domain pictures in insurance customer service scenarios, pseudo-annotate them through the OCR large model, extract key element fields in business domain pictures by combining manual annotation, convert the annotated data into a structured format, and complete the construction of the business domain picture dataset;

[0013] S4. Integrate the general domain image dataset and the business domain image dataset, perform further training based on the pre-trained OCR big model, and generate a multimodal insurance customer service big model optimized for insurance customer service scenarios;

[0014] S5. Deploy the insurance customer service multimodal large model to the GPU server.

[0015] Optionally, the S1 specifically includes the following steps:

[0016] S11. Obtain corpus data and font samples from public data sources and business data in insurance customer service scenarios, and match corpus and fonts:

[0017] ;

[0018] ;

[0019] in, represents a corpus collection, Represents the total amount of corpus data obtained from public data sources and insurance customer service business scenarios. Represents the total number of font samples used to generate the image. For a single corpus, and They are public data sources and insurance scenario business data sources. For font collections, For a single font, As a basic font sample, It is a font transformation function, including italic, bold or color adjustment;

[0020] S12. Use image generation tools to generate images based on the collected text corpus. and font samples Generate images with different fonts, sizes, colors, and backgrounds. The image generation rules include:

[0021] For a single line of text, generate an image with a maximum width not exceeding the preset pixels;

[0022] For multi-line text corpus, generate multi-line layout images, and use line breaks Indicates that paragraphs are wrapped with two line breaks express;

[0023] The background color set of the generated image is , where C represents the background color set, For the A specific color value;

[0024] S13, forming a picture-text pair P with the generated picture and the corresponding text;

[0025] S14, designing a prompt template, wherein the prompt template content includes picture content, text typesetting rules and key element extraction prompts, and combining the picture text pair P with the prompt template R to generate training data D;

[0026] S15. The generated training data D is trained by selecting a multimodal model based on MiniCPM-Llama3-V-2_5, and finally a preliminary OCR large model is obtained through training.

[0027] Optionally, the S15 specifically includes the following steps:

[0028] S151. Initialize the parameter set using the MiniCPM-Llama3-V-2_5 multimodal model , where W is the set of multimodal model weight matrices, including represents the image feature encoding weight, represents the text feature encoding weight, represents the multimodal fusion weight, represents the decoding weight, is the bias vector set;

[0029] Define the training optimization objective function , the training optimization objective function is based on the training data Build, where Represents the data input consisting of pictures and corresponding texts in the insurance customer service scenario. Representation and The corresponding standard output annotation is, is the number of training samples;

[0030] The identification and extraction of key elements are reflected by constructing a cross entropy loss function with multi-level nesting:

[0031] ;

[0032] in, is the number of feature categories, Indicates samples in the feature category The number of feature units under the feature unit, the feature unit includes text token or local area of ​​the image, is the actual annotation indicator value, used to identify the insurance customer service picture Sample No. Class The correct output of feature units, if the correct output , otherwise 0, Based on the current parameters and For Sample categories Next The predicted value of feature units, is an exponential function, is a logarithmic function;

[0033] S152, training data By batch size Divide into data batches , use the image encoding module to input the image Extract multimodal context features and transform the input image into the first Input images in batches The high-dimensional feature vector ;

[0034] S153. Text input The text encoding module is used for feature extraction, and the text is mapped into a text feature vector by performing word segmentation, embedding, and multi-head attention transformation on the input text sequence. ,The text feature vector fully reflects the text elements in the insurance customer service scenario, including name, policy number and claim amount;

[0035] S154, the image feature vector With text feature vector Through the multimodal fusion function To perform a joint representation:

[0036] ;

[0037] in, is the number of heads under the multi-head attention mechanism in the joint feature space, For the The number of feature units aggregated by the attention heads, It is the intermediate representation after weighted matching of image and text features with attention matrix. is the nonlinear transformation function after fusion, Indicates The attention head Integration parameters of feature units;

[0038] Through the joint representation process, the multimodal model can focus on both the key information in the image and the contextual meaning of the text in the insurance customer service scenario;

[0039] S155, using the decoding module to fusion features Make output predictions, map the fused representation to the target space, and generate prediction results ;

[0040] S156, using the back propagation algorithm to the objective function The gradient of is calculated to update the multimodal model parameters. For each training batch, the nested structure of chain rule and parameter derivation is used. and Execute parameter update:

[0041] ;

[0042] in, is the learning rate, which is dynamically set according to the difficulty of the insurance customer service multimodal data and the recognition accuracy requirements;

[0043] S157, repeat the steps S152 to S156 until the number of training rounds reaches the preset upper limit or the objective function converges, and the optimized parameters are Fixed, and got the preliminary OCR large model.

[0044] Optionally, the S2 specifically includes the following steps:

[0045] S21. Obtain real natural image data sets from multiple sources ,The image sources include payment screenshots, policy pictures and ID scans in the insurance customer service scenario;

[0046] S22. Use the pre-trained OCR model to analyze the real natural image data set Each picture in Perform text recognition and generate a set of image text results ;

[0047] S23, by designing prompt templates to collect picture text results Each text in Perform feature segmentation and annotation, and input the image text content and prompt template combination as follows:

[0048] ;

[0049] in, Prompt templates related to insurance customer service scenarios;

[0050] S24, input data The text is passed to the LLM-based text element extraction module to perform semantic analysis and key element identification on the text, and generate a set of key elements and corresponding values:

[0051] ;

[0052] Among them, the extraction of key elements is completed based on the following multi-level nesting:

[0053] ;

[0054] ;

[0055] in, Indicates from the picture No. A set of key elements extracted from a text segment, each of which includes a name and the corresponding values , For the In the picture The first paragraph of the text The predicted value of the key elements, Represents text The number of heads in the multi-head attention mechanism, Indicated in The number of feature units participating in the aggregation under the attention head is used to capture the local characteristics of the text and template prompts, is the activation function, and is the fusion weight of the attention mechanism and the semantic analysis module, is the multi-head attention calculation formula, is the query vector, is the key vector, is a value vector, is the key vector dimension, Represents a semantic analysis function based on parameters Extract text With template contextual features;

[0056] S25. For all pictures The key element extraction results Integrate and generate a general domain image dataset:

[0057] ;

[0058] in, For the The structured data of the image is in the form of:

[0059] ;

[0060] in, For the The total number of all key features in the image.

[0061] Optionally, S3 specifically includes the following steps:

[0062] S31, collecting business domain image data, and performing size normalization, rotation correction, and noise filtering on the business domain image data;

[0063] S32, using the pre-trained OCR large model to perform text recognition on the business domain image data, and extract text information from the business domain image data;

[0064] S33, combining the extracted text information with the prompt template by designing a prompt template related to the insurance customer service scenario to generate preliminary pseudo-annotated data;

[0065] S34, manually reviewing and annotating the pseudo-annotated data to extract key element fields in the text, and generating an annotated data set;

[0066] S35, performing data enhancement on the annotated data, and generating multiple versions of image data through rotation, blurring, and color adjustment operations;

[0067] S36. Integrate the annotated data and data enhancement content to generate business domain image datasets .

[0068] Optionally, the S4 specifically includes the following steps:

[0069] S41. General Domain Image Dataset Business domain image dataset According to the data volume ratio Sampling is performed, where is the scale factor of the common domain data, and is integrated to form a joint dataset:

[0070] ;

[0071] S42. For the joint dataset Perform batch division, set the batch size to B, and divide the joint dataset into a set of data batches, each batch contains general domain data and business domain data;

[0072] S43. Initialization parameters of the pre-trained OCR large model , set the optimization objective function to be a multimodal joint loss function :

[0073] ;

[0074] in:

[0075] ;

[0076] ;

[0077] in, is the cross entropy loss function, and are the predicted and true image feature representations, respectively. and are the predicted and true key element sets, is the image feature loss, is the text feature loss;

[0078] Set the initial learning rate , and train by combining the optimization objective function and model parameters;

[0079] S44, use cosine annealing learning rate scheduling strategy to adjust the learning rate :

[0080] ;

[0081] Among them, t is the current training step number, T is the total training step number;

[0082] S45. For each data batch , using the joint loss function Update model parameters through forward propagation and backpropagation;

[0083] S46, repeat S42 to S45 until the preset number of training rounds or the joint loss function is reached Convergence, and finally fix the optimized model parameters to generate a large multimodal model of insurance customer service optimized for insurance customer service scenarios.

[0084] Optionally, the S5 specifically includes the following steps:

[0085] S51, deploy the insurance customer service multimodal large model to the GPU server, and configure the server's operating environment, including graphics card driver, model reasoning framework, and network interface;

[0086] S52, receiving input image data through an open service interface, calling the insurance customer service multimodal large model to process the image, and outputting structured key information data to form a result data set;

[0087] S53, monitoring the operation status of the insurance customer service multimodal large model service, and establishing an operation data log library by recording call logs, including call time, processing time, input image information, and output results;

[0088] S54. Perform regular quality inspections on the output results and call logs of the insurance customer service multimodal big model, evaluate the performance of the insurance customer service multimodal big model in actual scenarios by setting data accuracy, processing efficiency, and user feedback indicators, and identify existing error cases;

[0089] S55. Collect and organize the error cases found during the quality inspection process into a new data set, and combine manual annotation to improve their key information to form extended data of error cases;

[0090] S56. Based on the expanded business domain data set, incremental training is performed on the existing insurance customer service multimodal large model, and steps S42 to S46 are repeatedly performed to update the parameters of the insurance customer service multimodal large model;

[0091] S57. Redeploy the optimized version of the insurance customer service multimodal big model to the server, replace the old version of the insurance customer service multimodal big model, and continue to provide services through the interface to complete the dynamic optimization and iterative upgrade of the insurance customer service multimodal big model.

[0092] The beneficial effects of the present invention are:

[0093] (1) The present invention uses a large multimodal model to deeply integrate image feature extraction and text element analysis, realizing an end-to-end processing method from images to structured data. The multimodal model training and optimization strategy solves the problem that traditional pipeline solutions require multiple independent modules to work together, significantly reducing the complexity and error risk of the system. The introduction of a unified multimodal model architecture can simultaneously process image content and text context semantics, avoiding the problem of overall result failure caused by sub-model error propagation in modular solutions.

[0094] (2) The present invention proposes a method for integrating and dynamically balancing general domain data and business domain data. Through data ratio sampling and batch division technology, it ensures that the model can evenly learn feature distribution on different types of images. Traditional methods cannot efficiently integrate data from multiple sources, which easily leads to insufficient model adaptation to a small number of key business scenarios. The dynamic balance training strategy of the present invention ensures that the model can adapt to complex general scenarios and insurance business domain scenarios at the same time, thereby improving the adaptability and scalability of the model.

[0095] (3) The present invention introduces a continuous optimization mechanism based on real-time calls and log feedback. It automatically collects and integrates the error cases identified by the model in actual scenarios into the business domain data set to form a closed-loop optimization process. Traditional static models lack self-optimization capabilities, while the dynamic feedback mechanism of the present invention enables the model to quickly adapt to new scenarios and changes in data distribution, thereby achieving continuous optimization and significantly improving the robustness and long-term stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0097] Figure 1 This is a flowchart of a method for structured extraction of text information from insurance customer service images based on a multimodal large model proposed by the present invention. DETAILED DESCRIPTION

[0098] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0099] refer to Figure 1 , a method for extracting structured text information from insurance customer service pictures based on a multimodal large model, comprising the following steps:

[0100] S1. Collect text corpus and font samples, and use multimodal training strategy to train the text corpus and font samples to obtain a preliminary OCR large model;

[0101] S2. Collect real natural images from multiple sources, use the OCR large model to recognize the text content in real natural images, and complete the construction of a general domain image dataset;

[0102] S3. Collect business domain pictures in insurance customer service scenarios, pseudo-annotate them through the OCR large model, extract key element fields in business domain pictures by combining manual annotation, convert the annotated data into a structured format, and complete the construction of the business domain picture dataset;

[0103] S4. Integrate the general domain image dataset and the business domain image dataset, perform further training based on the pre-trained OCR big model, and generate a multimodal insurance customer service big model optimized for insurance customer service scenarios;

[0104] S5. Deploy the insurance customer service multimodal large model to the GPU server.

[0105] In this implementation, S1 specifically includes the following steps:

[0106] S11. Obtain corpus data and font samples from public data sources and business data in insurance customer service scenarios, and match corpus and fonts:

[0107] ;

[0108] ;

[0109] in, represents a corpus collection, Represents the total amount of corpus data obtained from public data sources and insurance customer service business scenarios. Represents the total number of font samples used to generate the image. For a single corpus, and They are public data sources and insurance scenario business data sources. For font collections, For a single font, As a basic font sample, It is a font transformation function, including italic, bold or color adjustment;

[0110] S12. Use image generation tools to generate images based on the collected text corpus. and font samples Generate images with different fonts, sizes, colors, and backgrounds. The image generation rules include:

[0111] For a single line of text, generate an image with a maximum width not exceeding the preset pixels;

[0112] For multi-line text corpus, generate multi-line layout images, and use line breaks Indicates that paragraphs are wrapped with two line breaks express;

[0113] The background color set of the generated image is , where C represents the background color set, For the A specific color value;

[0114] S13, forming a picture-text pair P with the generated picture and the corresponding text;

[0115] S14, designing a prompt template, wherein the prompt template content includes picture content, text typesetting rules and key element extraction prompts, and combining the picture text pair P with the prompt template R to generate training data D;

[0116] S15. The generated training data D is trained by selecting a multimodal model based on MiniCPM-Llama3-V-2_5, and finally a preliminary OCR large model is obtained through training.

[0117] In this implementation, S15 specifically includes the following steps:

[0118] S151. Initialize the parameter set using the MiniCPM-Llama3-V-2_5 multimodal model , where W is the set of multimodal model weight matrices, including represents the image feature encoding weight, represents the text feature encoding weight, represents the multimodal fusion weight, represents the decoding weight, is the bias vector set;

[0119] Define the training optimization objective function , the training optimization objective function is based on the training data Build, where Represents the data input consisting of pictures and corresponding texts in the insurance customer service scenario. Representation and The corresponding standard output annotation is, is the number of training samples;

[0120] The identification and extraction of key elements are reflected by constructing a cross entropy loss function with multi-level nesting:

[0121] ;

[0122] in, is the number of feature categories, Indicates samples in the feature category The number of feature units under the feature unit, the feature unit includes text token or local area of ​​the image, is the actual annotation indicator value, used to identify the insurance customer service picture Sample No. Class The correct output of feature units, if the correct output , otherwise 0, Based on the current parameters and For Sample categories Next The predicted value of feature units, is an exponential function, is a logarithmic function;

[0123] S152, training data By batch size Divide into data batches , use the image encoding module to input the image Extract multimodal context features and transform the input image into the first Input images in batches The high-dimensional feature vector ;

[0124] S153. Text input The text encoding module is used for feature extraction, and the text is mapped into a text feature vector by performing word segmentation, embedding, and multi-head attention transformation on the input text sequence. ,The text feature vector fully reflects the text elements in the insurance customer service scenario, including name, policy number and claim amount;

[0125] S154, the image feature vector With text feature vector Through the multimodal fusion function To perform a joint representation:

[0126] ;

[0127] in, is the number of heads under the multi-head attention mechanism in the joint feature space, For the The number of feature units aggregated by the attention heads, It is the intermediate representation after weighted matching of image and text features with attention matrix. is the nonlinear transformation function after fusion, Indicates The attention head Integration parameters of feature units;

[0128] Through the joint representation process, the multimodal model can focus on both the key information in the image and the contextual meaning of the text in the insurance customer service scenario;

[0129] S155, using the decoding module to fusion features Make output predictions, map the fused representation to the target space, and generate prediction results ;

[0130] S156, using the back propagation algorithm to the objective function The gradient of is calculated to update the multimodal model parameters. For each training batch, the nested structure of chain rule and parameter derivation is used. and Execute parameter update:

[0131] ;

[0132] in, is the learning rate, which is dynamically set according to the difficulty of the insurance customer service multimodal data and the recognition accuracy requirements;

[0133] S157, repeat the steps S152 to S156 until the number of training rounds reaches the preset upper limit or the objective function converges, and the optimized parameters are Fixed, and got the preliminary OCR large model.

[0134] In this implementation, S2 specifically includes the following steps:

[0135] S21. Obtain real natural image data sets from multiple sources ,The image sources include payment screenshots, policy pictures and ID scans in the insurance customer service scenario;

[0136] S22. Use the pre-trained OCR model to analyze the real natural image data set Each picture in Perform text recognition and generate a set of image text results ;

[0137] S23, by designing prompt templates to collect picture text results Each text in Perform feature segmentation and annotation, and input the image text content and prompt template combination as follows:

[0138] ;

[0139] in, Prompt templates related to insurance customer service scenarios;

[0140] S24, input data The text is passed to the LLM-based text element extraction module to perform semantic analysis and key element identification on the text, and generate a set of key elements and corresponding values:

[0141] ;

[0142] Among them, the extraction of key elements is completed based on the following multi-level nesting:

[0143] ;

[0144] ;

[0145] in, Indicates from the picture No. A set of key elements extracted from a text segment, each of which includes a name and the corresponding values , For the In the picture The first paragraph of the text The predicted value of the key elements, Represents text The number of heads in the multi-head attention mechanism, Indicated in The number of feature units participating in the aggregation under the attention head is used to capture the local characteristics of the text and template prompts, is the activation function, and is the fusion weight of the attention mechanism and the semantic analysis module, is the multi-head attention calculation formula, is the query vector, is the key vector, is a value vector, is the key vector dimension, Represents a semantic analysis function based on parameters Extract text With template contextual features;

[0146] S25. For all pictures The key element extraction results Integrate and generate a general domain image dataset:

[0147] ;

[0148] in, For the The structured data of the image is in the form of:

[0149] ;

[0150] in, For the The total number of all key features in the image.

[0151] In this implementation, S3 specifically includes the following steps:

[0152] S31, collecting business domain image data, and performing size normalization, rotation correction, and noise filtering on the business domain image data;

[0153] S32, using the pre-trained OCR large model to perform text recognition on the business domain image data, and extract text information from the business domain image data;

[0154] S33, combining the extracted text information with the prompt template by designing a prompt template related to the insurance customer service scenario to generate preliminary pseudo-annotated data;

[0155] S34, manually reviewing and annotating the pseudo-annotated data to extract key element fields in the text, and generating an annotated data set;

[0156] S35, performing data enhancement on the annotated data, and generating multiple versions of image data through rotation, blurring, and color adjustment operations;

[0157] S36. Integrate the annotated data and data enhancement content to generate business domain image datasets .

[0158] In this implementation, S4 specifically includes the following steps:

[0159] S41. General Domain Image Dataset Business domain image dataset According to the data volume ratio Sampling is performed, where is the scale factor of the common domain data, and is integrated to form a joint dataset:

[0160] ;

[0161] S42. For the joint dataset Perform batch division, set the batch size to B, and divide the joint dataset into a set of data batches, each batch contains general domain data and business domain data;

[0162] S43. Initialization parameters of the pre-trained OCR large model , set the optimization objective function to be a multimodal joint loss function :

[0163] ;

[0164] in:

[0165] ;

[0166] ;

[0167] in, is the cross entropy loss function, and are the predicted and true image feature representations, respectively. and are the predicted and true key element sets, is the image feature loss, is the text feature loss;

[0168] Set the initial learning rate , and train by combining the optimization objective function and model parameters;

[0169] S44, use cosine annealing learning rate scheduling strategy to adjust the learning rate :

[0170] ;

[0171] Among them, t is the current training step number, T is the total training step number;

[0172] S45. For each data batch , using the joint loss function Update model parameters through forward propagation and backpropagation;

[0173] S46, repeat S42 to S45 until the preset number of training rounds or the joint loss function is reached Convergence, and finally fix the optimized model parameters to generate a large multimodal model of insurance customer service optimized for insurance customer service scenarios.

[0174] In this implementation, S5 specifically includes the following steps:

[0175] S51, deploy the insurance customer service multimodal large model to the GPU server, and configure the server's operating environment, including graphics card driver, model reasoning framework, and network interface;

[0176] S52, receiving input image data through an open service interface, calling the insurance customer service multimodal large model to process the image, and outputting structured key information data to form a result data set;

[0177] S53, monitoring the operation status of the insurance customer service multimodal large model service, and establishing an operation data log library by recording call logs, including call time, processing time, input image information, and output results;

[0178] S54. Perform regular quality inspections on the output results and call logs of the insurance customer service multimodal big model, evaluate the performance of the insurance customer service multimodal big model in actual scenarios by setting data accuracy, processing efficiency, and user feedback indicators, and identify existing error cases;

[0179] S55. Collect and organize the error cases found during the quality inspection process into a new data set, and combine manual annotation to improve their key information to form extended data of error cases;

[0180] S56. Based on the expanded business domain data set, incremental training is performed on the existing insurance customer service multimodal large model, and steps S42 to S46 are repeatedly performed to update the parameters of the insurance customer service multimodal large model;

[0181] S57. Redeploy the optimized version of the insurance customer service multimodal big model to the server, replace the old version of the insurance customer service multimodal big model, and continue to provide services through the interface to complete the dynamic optimization and iterative upgrade of the insurance customer service multimodal big model.

[0182] Embodiment 1:

[0183] The implementation example is applied in the daily operation of the insurance customer service center, which needs to process a large number of document images uploaded by customers every day, including payment screenshots, policy scans, ID photos, and driving license photos. The pictures often contain key business information, such as customer name, policy number, and payment amount. The information needs to be extracted and structured quickly and accurately for efficient processing of subsequent business. However, traditional pipeline technology has obvious limitations in actual scenarios, such as low recognition accuracy, high system complexity caused by multi-model collaboration, and insufficient adaptability to new scenarios.

[0184] A large insurance company needs to process claim application images submitted by customers, mainly including payment screenshots and policy images. In actual cases, the number of images uploaded in one day can reach more than 5,000, of which nearly 30% have complex formats, including payment screenshots with multiple amount fields and blurred separators, and policy images with irregular content layout. The traditional pipeline solution uses image classification models, OCR character recognition models, and structured logic rules to gradually process images. Due to the complexity of the system and the strong dependence between each link, the processing efficiency is extremely low, and the recognition error rate is as high as 18%.

[0185] To solve this problem, the company decided to apply the structured extraction method of insurance customer service image text information based on a multimodal large model of the present invention to build an efficient and intelligent image information extraction system to improve recognition accuracy and overall processing efficiency.

[0186] First, we built a dataset. Based on the general domain image dataset and the business domain image dataset, we collected 2 million historical images for insurance customer service scenarios, including 1 million payment screenshots, 500,000 insurance policy images, and the rest were ID card and driving license images. We preprocessed the data according to the data processing flow, including size normalization, text recognition pseudo-annotation and manual annotation, and generated multiple versions of data through data enhancement technology, ultimately forming a training dataset of 3 million images.

[0187] Subsequently, the multimodal large model was optimized and trained, and the ratio of general domain and business domain image data was sampled at 7:3. The model used MiniCPM-Llama3-V-2_5 as the base. After 120 rounds of training, the text recognition accuracy of the model was significantly improved. In the test set, the model's correct recognition rate of the amount separator in the payment screenshot was increased from 82% of the traditional solution to 96%, and the accuracy of extracting the policy number in the policy image was increased from 85% of the traditional solution to 98%.

[0188] After the system was deployed, it was tested for a week in a customer service center. Among the 5,000 images processed every day, the method of the present invention was able to complete the recognition and structuring operations at an average speed of 0.8 seconds per image, while the traditional method took 3 seconds. Through log feedback and manual quality inspection, it was found that the overall recognition accuracy of the method of the present invention reached 97.5%, while the traditional method was only 84%. For newly emerging formats, such as multi-amount fields in specific bank payment screenshots, the present invention quickly optimizes the model through incremental training, maintaining high adaptability to new scenarios.

[0189] Specific example pictures:

[0190] Payment screenshot sample 1: The traditional method cannot correctly identify the error caused by the fuzzy separator in the amount field, and the extraction result is "￥1.23,000", while the method of the present invention successfully extracts "￥123,000".

[0191] Insurance policy image sample 2: The traditional method makes an error in extracting the irregularly formatted insurance policy number, resulting in a null value, while the method of the present invention successfully extracts the insurance policy number "PN123456789".

[0192] It can be seen from this embodiment that the structured extraction method of insurance customer service image text information based on a multimodal large model of the present invention not only significantly improves the efficiency and accuracy of image text information extraction, but also reduces the complexity of the system, and the adaptability to new scenarios is also enhanced. The present invention demonstrates significant superiority in complex business scenarios.

[0193] The present invention adopts a large multimodal model to deeply integrate image feature extraction and text element analysis, realizing an end-to-end processing method from images to structured data. The multimodal model training and optimization strategy solves the problem that the traditional pipeline solution requires multiple independent modules to work together, significantly reducing the complexity and error risk of the system. The introduction of a unified multimodal model architecture can simultaneously process image content and text context semantics, avoiding the problem of overall result failure caused by sub-model error propagation in modular solutions.

[0194] The present invention proposes an integration and dynamic balance training method for general domain data and business domain data. Through data ratio sampling and batch division technology, it ensures that the model can evenly learn feature distribution on different types of images. Traditional methods cannot efficiently integrate data from multiple sources, which easily leads to insufficient model adaptation to a small number of key business scenarios. The dynamic balance training strategy of the present invention ensures that the model can adapt to complex general scenarios and insurance business domain scenarios at the same time, thereby improving the adaptability and scalability of the model.

[0195] The present invention introduces a continuous optimization mechanism based on real-time calls and log feedback, which automatically collects and integrates error cases identified by the model in actual scenarios into the business domain data set to form a closed-loop optimization process. Traditional static models lack self-optimization capabilities, and the dynamic feedback mechanism of the present invention enables the model to quickly adapt to new scenarios and changes in data distribution, thereby achieving continuous optimization and significantly improving the robustness and long-term stability of the system.

[0196] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for extracting structured text information from insurance customer service images based on a multimodal large model, characterized in that: The steps include: S1. Collect text corpus and font samples, and use multimodal training strategy to train the text corpus and font samples to obtain a preliminary OCR large model; S2. Collect real natural images from multiple sources, use the OCR large model to recognize the text content in real natural images, and complete the construction of a general domain image dataset; S3. Collect business domain pictures in insurance customer service scenarios, pseudo-annotate them through the OCR large model, extract key element fields in business domain pictures by combining manual annotation, convert the annotated data into a structured format, and complete the construction of the business domain picture dataset; S4. Integrate the general domain image dataset and the business domain image dataset, perform further training based on the pre-trained OCR big model, and generate a multimodal insurance customer service big model optimized for insurance customer service scenarios; The S4 specifically comprises the following steps: S41. General Domain Image Dataset Business domain image dataset According to the data volume ratio Sampling is performed, where is the scale factor of the common domain data, and is integrated to form a joint dataset: ； S42. For the joint dataset Perform batch division, set the batch size to B, and divide the joint dataset into a set of data batches, each batch contains general domain data and business domain data; S43. Initialization parameters of the pre-trained OCR large model , set the optimization objective function to be a multimodal joint loss function : ； in: ； ； in, is the cross entropy loss function, and are the predicted and true image feature representations, respectively. and are the sets of predicted and real key elements, respectively. is the image feature loss, is the text feature loss; Set the initial learning rate , and train by combining the optimization objective function and model parameters; S44, use cosine annealing learning rate scheduling strategy to adjust the learning rate : ； Among them, t is the current training step number, is the total number of training steps; S45. For each data batch , using the joint loss function Update model parameters through forward propagation and backpropagation; S46, repeat S42 to S45 until the preset number of training rounds or the joint loss function is reached Convergence, finally fix the optimized model parameters, and generate a large multi-modal insurance customer service model optimized for insurance customer service scenarios; S5. Deploy the insurance customer service multimodal large model to the GPU server.

2. According to claim 1, a method for extracting structured text information from insurance customer service pictures based on a multimodal large model is characterized in that: The S1 specifically includes the following steps: S11. Obtain corpus data and font samples from public data sources and business data in insurance customer service scenarios, and match corpus and fonts: ； ； in, represents a corpus collection, Represents the total amount of corpus data obtained from public data sources and insurance customer service business scenarios. Represents the total number of font samples used to generate the image. For a single corpus, and They are public data sources and insurance scenario business data sources. For font collections, For a single font, As a basic font sample, It is a font transformation function, including italic, bold or color adjustment; S12. Use image generation tools to generate images based on the collected text corpus. and font samples Generate images with different fonts, sizes, colors, and backgrounds. The image generation rules include: For a single line of text, generate an image with a maximum width not exceeding the preset pixels; For multi-line text corpus, generate multi-line layout images, and use line breaks Indicates that paragraphs are wrapped with two line breaks express; The background color set of the generated image is , where C represents the background color set, For the A specific color value; S13, forming a picture-text pair P with the generated picture and the corresponding text; S14, designing a prompt template, wherein the prompt template content includes picture content, text typesetting rules and key element extraction prompts, and combining the picture text pair P with the prompt template R to generate training data D; S15. The generated training data D is trained by selecting a multimodal model based on MiniCPM-Llama3-V-2_5, and finally a preliminary OCR large model is obtained through training.

3. According to claim 2, a method for extracting structured text information from insurance customer service pictures based on a multimodal large model is characterized in that: The S15 specifically includes the following steps: S151. Initialize the parameter set using the MiniCPM-Llama3-V-2_5 multimodal model , where W is the set of multimodal model weight matrices, including represents the image feature encoding weight, represents the text feature encoding weight, represents the multimodal fusion weight, represents the decoding weight, is the bias vector set; Define the training optimization objective function , the training optimization objective function is based on the training data Build, where Represents the data input consisting of pictures and corresponding texts in the insurance customer service scenario. Representation and The corresponding standard output annotation is, is the number of training samples; The identification and extraction of key elements are reflected by constructing a cross entropy loss function with multi-level nesting: ； in, is the number of feature categories, Indicates samples in the feature category The number of feature units under the feature unit, the feature unit includes text token or local area of ​​the image, is the actual annotation indicator value, used to identify the insurance customer service picture Sample No. Class The correct output of feature units, if the correct output , otherwise 0, Based on the current parameters and For Sample categories Next The predicted value of feature units, is an exponential function, is a logarithmic function; S152, training data By batch size Divide into data batches , use the image encoding module to input the image Extract multimodal context features and transform the input image into the first Input images in batches The high-dimensional feature vector ; S153. Text input The text encoding module is used for feature extraction, and the text is mapped into a text feature vector by performing word segmentation, embedding, and multi-head attention transformation on the input text sequence. ,The text feature vector fully reflects the text elements in the insurance customer service scenario, including name, policy number and claim amount; S154, the image feature vector With text feature vector Through the multimodal fusion function To perform a joint representation: ； in, is the number of heads under the multi-head attention mechanism in the joint feature space, For the The number of feature units aggregated by the attention heads, It is the intermediate representation after weighted matching of image and text features with attention matrix. is the nonlinear transformation function after fusion, Indicates The attention head Integration parameters of feature units; Through the joint representation process, the multimodal model can focus on both the key information in the image and the contextual meaning of the text in the insurance customer service scenario; S155, using the decoding module to fusion features Make output predictions, map the fused representation to the target space, and generate prediction results ; S156, using the back propagation algorithm to the objective function The gradient of is calculated to update the multimodal model parameters. For each training batch, the nested structure of chain rule and parameter derivation is used. and Execute parameter update: ； in, is the learning rate, which is dynamically set according to the difficulty of the insurance customer service multimodal data and the recognition accuracy requirements; S157, repeat the steps S152 to S156 until the number of training rounds reaches the preset upper limit or the objective function converges, and the optimized parameters are Fixed, and got the preliminary OCR large model.

4. According to claim 1, a method for extracting structured text information from insurance customer service pictures based on a multimodal large model is characterized in that: The S2 specifically includes the following steps: S21. Obtain real natural image data sets from multiple sources ,The image sources include payment screenshots, policy pictures and ID scans in the insurance customer service scenario; S22. Use the pre-trained OCR model to analyze the real natural image data set Each picture in Perform text recognition and generate a set of image text results ; S23, by designing prompt templates to collect picture text results Each text in Perform feature segmentation and annotation, and input the image text content and prompt template combination as follows: ； in, Prompt templates related to insurance customer service scenarios; S24, input data The text is passed to the LLM-based text element extraction module to perform semantic analysis and key element identification on the text, and generate a set of key elements and corresponding values: ； Among them, the extraction of key elements is completed based on the following multi-level nesting: ； ； in, Indicates from the picture No. A set of key elements extracted from a text segment, each of which includes a name and the corresponding values , For the In the picture The first paragraph of the text The predicted value of the key elements, Represents text The number of heads in the multi-head attention mechanism, Indicated in The number of feature units participating in the aggregation under the attention head is used to capture the local characteristics of the text and template prompts, is the activation function, and is the fusion weight of the attention mechanism and the semantic analysis module, is the multi-head attention calculation formula, is the query vector, is the key vector, is a value vector, is the key vector dimension, Represents a semantic analysis function based on parameters Extract text With template contextual features; S25. For all pictures The key elements extraction results Integrate and generate a general domain image dataset: ； in, For the The structured data of the image is as follows: ； in, For the The total number of all key features in the image.

5. The method for extracting structured text information from insurance customer service pictures based on a multimodal large model according to claim 1 is characterized in that: The S3 specifically includes the following steps: S31, collecting business domain image data, and performing size normalization, rotation correction, and noise filtering on the business domain image data; S32, using the pre-trained OCR large model to perform text recognition on the business domain image data, and extract text information from the business domain image data; S33, combining the extracted text information with the prompt template by designing a prompt template related to the insurance customer service scenario to generate preliminary pseudo-annotated data; S34, manually reviewing and annotating the pseudo-annotated data to extract key element fields in the text, and generating an annotated data set; S35, performing data enhancement on the annotated data, and generating multiple versions of image data through rotation, blurring, and color adjustment operations; S36. Integrate the annotated data and data enhancement content to generate business domain image datasets .

6. The method for extracting structured text information from insurance customer service pictures based on a multimodal large model according to claim 1 is characterized in that: The S5 specifically includes the following steps: S51, deploy the insurance customer service multimodal large model to the GPU server, and configure the server's operating environment, including graphics card driver, model reasoning framework, and network interface; S52, receiving input image data through an open service interface, calling the insurance customer service multimodal large model to process the image, and outputting structured key information data to form a result data set; S53, monitoring the operation status of the insurance customer service multimodal large model service, and establishing an operation data log library by recording call logs, including call time, processing time, input image information, and output results; S54. Perform regular quality inspections on the output results and call logs of the insurance customer service multimodal big model, evaluate the performance of the insurance customer service multimodal big model in actual scenarios by setting data accuracy, processing efficiency, and user feedback indicators, and identify existing error cases; S55. Collect and organize the error cases found during the quality inspection process into a new data set, and combine manual annotation to improve their key information to form extended data of error cases; S56. Based on the expanded business domain data set, incremental training is performed on the existing insurance customer service multimodal large model, and steps S42 to S46 are repeatedly performed to update the parameters of the insurance customer service multimodal large model; S57. Redeploy the optimized version of the insurance customer service multimodal big model to the server, replace the old version of the insurance customer service multimodal big model, and continue to provide services through the interface to complete the dynamic optimization and iterative upgrade of the insurance customer service multimodal big model.

Images