Method, apparatus, and storage medium for processing picture information

Abstract

The application discloses a picture information processing method and device, electronic equipment and a storage medium, and relates to the technical field of data processing. A specific embodiment of the method comprises: acquiring a preset basic large model and a first sample set, training the basic large model based on the first sample set to obtain an initial model; acquiring a preset second sample set, inputting the initial model to obtain an output result, and checking the output result; in response to the checking result not satisfying a preset condition, adjusting the initial model, inputting the second sample set into the adjusted initial model, obtaining an updated output result and checking the same, and obtaining a feature extraction model when a new checking result satisfies the preset condition; and in response to a picture feature extraction instruction, acquiring a picture to be processed and inputting the same into the feature extraction model to obtain a structured picture feature. The embodiment can solve the problem that the mode of first obtaining a text line and then extracting structured information is sensitive to layout changes and the accuracy of information extraction is low in a complex scene.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, electronic device, and storage medium for image information processing. Background Technology

[0002] Image information extraction refers to the automatic identification and output of standardized data that is machine-readable, database-compatible, and computationally calculable from images containing text, tables, layouts, symbols, etc. In business scenarios involving invoices, certificates, contracts, etc., image structured information extraction is a common method. Related technologies can first obtain text lines with coordinates through text detection, and then extract structured information from these text lines based on rules and templates. However, this method is sensitive to changes in image layout, and its accuracy is low in complex scenarios such as complex layouts and blurry images. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a method, apparatus, electronic device, and storage medium for image information processing, which can solve the problem that the method of first obtaining text lines with coordinates and then extracting structured information from the text lines is sensitive to layout changes and has low accuracy in information extraction in complex scenarios such as complex layouts and blurry images.

[0004] To achieve the above objectives, according to one aspect of the present invention, a method for image information processing is provided.

[0005] An image information processing method according to an embodiment of the present invention includes: obtaining a preset basic large model and a first sample set, calling a first training algorithm, training the basic large model based on the first sample set, and obtaining an initial model, wherein the first sample set includes pre-labeled image samples; Obtain a preset second sample set, input it into the initial model, obtain the output result, and verify the output result; In response to the verification result not meeting the preset conditions, the initial model is adjusted based on the verification result and the preset optimization strategy. The second sample set is input into the adjusted initial model to obtain an updated output result, which is then verified to obtain a new verification result. If the new verification result meets the preset conditions, the adjusted initial model is determined as the feature extraction model. In response to an image feature extraction instruction, an image to be processed is acquired and input into the feature extraction model to obtain structured image features.

[0006] In one embodiment, before obtaining the preset base large model and the first sample set, the following steps are included: Collect the first set of images associated with the preset business type; For each image in the first image set, identify the image elements of the image, determine the feature extraction type of the image based on the image elements, and generate annotation information corresponding to the feature extraction type; The annotation information of each image in the first image set is verified, and an image sample set is determined from the first image set based on the verification results to generate the first sample set.

[0007] In yet another embodiment, identifying image elements of the image and determining the feature extraction type of the image based on the image elements includes: Identify the image elements of the image, classify the image elements, and determine the types of elements included in the image; The feature extraction type of the image is determined based on the element type.

[0008] In yet another embodiment, generating the first sample set includes: For each image in the image sample set, the corresponding corpus is matched from the business corpus, the corresponding script template is queried from the corpus, and the instruction text corresponding to the image is generated based on the script template. The business corpus is constructed from voice consultation records, customer service chat records, and comment records associated with preset business types. The first sample set is generated based on the annotation information and instruction text of each image in the image sample set.

[0009] In yet another embodiment, before obtaining a preset second sample set, the process includes: Obtain the training data corresponding to the initial model, and extract a second set of images from the training data whose deviation between the predicted result and the actual result is greater than a preset threshold. The preset noise overlay strategies are invoked to overlay noise onto each image in the second image set, resulting in the second sample set.

[0010] In yet another embodiment, the initial model includes a visual feature extraction layer; Before adjusting the initial model based on the verification results and the preset optimization strategy, the process also includes: Obtain the model parameters corresponding to the visual feature extraction layer in the initial model, and update the preset parameters of the model parameters to the target values; Adjusting the initial model based on the verification results and a preset optimization strategy includes: Obtain the model parameters in the initial model, filter the model parameters to be adjusted that are not the target values ​​of the preset parameters, and adjust the model parameters to be adjusted based on the verification results and the preset optimization strategy.

[0011] To achieve the above objectives, according to another aspect of the present invention, an apparatus for image information processing is provided.

[0012] An image information processing apparatus according to an embodiment of the present invention includes: a training unit, configured to acquire a preset basic large model and a first sample set, invoke a first training algorithm, train the basic large model based on the first sample set, and obtain an initial model, wherein the first sample set includes pre-labeled image samples; The training unit is also used to obtain a preset second sample set, input it into the initial model, obtain the output result, and verify the output result; An adjustment unit is used to adjust the initial model based on the verification result and a preset optimization strategy in response to the verification result not meeting the preset conditions. The second sample set is input into the adjusted initial model to obtain an updated output result and perform verification to obtain a new verification result. If the new verification result meets the preset conditions, the adjusted initial model is determined as the feature extraction model. The extraction unit is used to respond to the image feature extraction instruction, acquire the image to be processed, input it into the feature extraction model, and output structured image features.

[0013] In one embodiment, the apparatus further includes: The acquisition unit is used to acquire a first set of images associated with a preset business type; The generation unit is used to identify the image elements of each image in the first image set, determine the feature extraction type of the image based on the image elements, and generate annotation information corresponding to the feature extraction type. The generation unit is further configured to verify each image in the first image set and the annotation information of each image, and determine an image sample set from the first image set based on the verification result, so as to generate the first sample set.

[0014] In yet another embodiment, the generating unit is specifically used for: Identify the image elements of the image, classify the image elements, and determine the types of elements included in the image; The feature extraction type of the image is determined based on the element type.

[0015] In yet another embodiment, the generating unit is specifically used for: For each image in the image sample set, the corresponding corpus is matched from the business corpus, the corresponding script template is queried from the corpus, and the instruction text corresponding to the image is generated based on the script template. The business corpus is constructed from voice consultation records, customer service chat records, and comment records associated with preset business types. The first sample set is generated based on the annotation information and instruction text of each image in the image sample set.

[0016] In yet another embodiment, the extraction unit is further configured to: Obtain the training data corresponding to the initial model, and extract a second set of images from the training data whose deviation between the predicted result and the actual result is greater than a preset threshold. The preset noise overlay strategies are invoked to overlay noise onto each image in the second image set, resulting in the second sample set.

[0017] In yet another embodiment, the initial model includes a visual feature extraction layer; The device further includes: The update unit is used to obtain the model parameters corresponding to the visual feature extraction layer in the initial model and update the preset parameters of the model parameters to the target value. The adjustment unit is specifically used for: Obtain the model parameters in the initial model, filter the model parameters to be adjusted that are not the target values ​​of the preset parameters, and adjust the model parameters to be adjusted based on the verification results and the preset optimization strategy.

[0018] To achieve the above objectives, according to another aspect of the present invention, an electronic device is provided.

[0019] An electronic device according to an embodiment of the present invention includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the image information processing method provided in the embodiment of the present invention.

[0020] To achieve the above objectives, according to another aspect of the present invention, a computer-readable medium is provided.

[0021] An embodiment of the present invention provides a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the image information processing method provided in the embodiment of the present invention.

[0022] To achieve the above objectives, according to another aspect of the present invention, a computer program product is provided.

[0023] A computer program product according to an embodiment of the present invention includes a computer program that, when executed by a processor, implements the image information processing method provided in the embodiment of the present invention.

[0024] One embodiment of the above invention has the following advantages or beneficial effects: In this embodiment of the invention, the basic large model can first be trained using a pre-labeled first sample set to obtain an initial model. Then, the initial model can be adjusted based on a second sample set to obtain a feature extraction model. In response to an image feature extraction command, the trained feature extraction model can process the image to be processed, yielding the corresponding structured image features. In this embodiment, the basic large model is trained in two stages. The first stage uses a pre-labeled sample set for training, enabling the initial model to quickly grasp the visual structure, semantic relationships, and basic structured output capabilities of the image, thus giving the initial model higher adaptability. The second stage adjusts the initial model based on an optimization strategy, resulting in a feature extraction model with higher accuracy. Therefore, the feature extraction model has higher adaptability and precision. Thus, by using the trained feature extraction model to process the structured image features of the image to be processed, the extraction results have higher accuracy.

[0025] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0026] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein: Figure 1 This is a schematic diagram of a main flow of an image information processing method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a model training process according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the main units of an image information processing apparatus according to an embodiment of the present invention; Figure 4 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied; Figure 5 This is a schematic diagram of the structure of a computer system suitable for implementing embodiments of the present invention. Detailed Implementation

[0027] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0028] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined with each other. The acquisition, transmission, storage, use, and processing of data in this application comply with relevant national laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0029] This invention provides an image information processing system that can be used in scenarios involving image information extraction, specifically in scenarios where features are extracted from images using a pre-trained model.

[0030] This invention provides a method for image information processing, which can be executed by an image information processing system, such as... Figure 1 As shown, the method includes the following steps.

[0031] S101: Obtain the preset basic large model and the first sample set, call the first training algorithm, train the basic large model based on the first sample set, and obtain the initial model.

[0032] The first sample set includes pre-labeled image samples. The base model can include various types of large models, such as a pre-trained multimodal large model. The first sample set is pre-generated and can include a set of image samples, each of which is pre-labeled. The base model can be trained using the first sample set via SFT (Supervised Fine-Tuning) training. This involves using the pre-trained multimodal large model as a foundation and supervising it with high-quality SFT-constructed image samples for multi-task training. This allows the multimodal large model to converge from general visual language understanding capabilities to specialized capabilities for extracting structured information (features) from images for a pre-defined business scenario. After training, an initial model is obtained.

[0033] Specifically, the first sample set can represent the input suitable for the basic large-scale model, and then input it into the basic large-scale model for training. In this embodiment of the invention, the format of the model input can be preset, and then the input of the preset format can be constructed based on the pre-labeled image samples in the first sample set, thereby improving the model's ability to understand and follow human intentions in preset business scenarios. The preset format can include: images, instructions, and structured output. Images represent pre-labeled image samples; instructions represent standardized instruction text, which can include task descriptions and output requirements, clearly specifying specific tasks in natural language, and can explicitly specify output forms such as concise expression, structured presentation, list listing, tabular display, or compliant response, such as: "Identify what this material is, extract the ID number and name, output a structured list, and do not disclose irrelevant information"; structured output represents a structured standard answer format, such as JSON, two-dimensional table, entity list, etc. In this way, images, standardized instructions, and output formats are multimodally aligned to form a unified sample input for training the basic large-scale model. In this embodiment of the invention, the format of the structured output can be preset to ensure that the output results are clear, standardized, and meet the usage requirements of the business scenario.

[0034] During training, multi-dimensional supervised loss joint optimization can be employed. The loss function can include minimizing the cross-entropy loss between the model's output sequence and the standard answer, facilitating the learning of field content, semantic relationships, and output logic in the image. In this embodiment, a low learning rate fine-tuning strategy can also be used to retain the general capabilities of the basic large model while quickly adapting to preset business scenarios. Depending on computing power and data scale, full-parameter fine-tuning or efficient parameter fine-tuning (such as LoRA) can be selected, focusing on optimizing the visual projection layer, multimodal fusion layer, and output layer. Batch training, gradient accumulation, and dynamic batch processing can also be used to ensure training stability and avoid catastrophic forgetting. Balanced sampling according to preset business scenarios ensures a balanced distribution of samples of all types, preventing the model from biased towards a particular data type. Model training can be continuously iterated within a preset number of rounds until the accuracy of the structured output fields reaches a stable high level, the format compliance rate approaches 100%, and the loss curve becomes relatively stable. This ensures that the resulting initial model can be a multimodal large model with basic structured information extraction capabilities, capable of performing tasks such as content recognition, field extraction, form parsing, and interface understanding in preset business scenarios, serving as a base model for subsequent stages.

[0035] In one implementation, the first sample set may be pre-constructed. To make the data in the sample set more comprehensive and improve training efficiency, this embodiment of the invention may collect image sets of various types corresponding to preset services and label them. Specifically, this can be performed as follows: collecting a first image set associated with a preset service type; for each image in the first image set, identifying the image elements of the image, determining the feature extraction type of the image based on the image elements, and generating labeling information corresponding to the feature extraction type; verifying each image and its labeling information in the first image set, and determining an image sample set from the first image set based on the verification results to generate the first sample set.

[0036] The collected images can cover various image scenarios and include richly annotated structured information extraction task datasets. This allows the model training to fully learn the spatial structure, layout relationships, entity attributes, and semantic associations within the images, outputting a finely tuned multimodal large model with basic structured information extraction capabilities. Taking government service as an example, the collected images need to achieve comprehensive and balanced coverage of core government service scenarios, including high-frequency business-related scenarios and special scenarios, so that the trained model has broad generalization capabilities for government service scenarios. Specifically, relevant government service scenarios can include: government materials and certificates scenarios, such as common certificates like ID cards, household registration books, business licenses, real estate certificates, social security cards, and driver's licenses, as well as various application forms, commitment letters, supporting documents, scanned copies of contracts, invoices, and other application materials; and government service interface scenarios, including online processing interfaces such as the one-stop online service page, business application page, material upload page, progress query page, and electronic signature page. Image acquisition can be achieved in various ways. For example, high-quality images can be selected and collected from open-source datasets, or targeted data acquisition and synthesis can be carried out for specific business scenarios. For instance, a VLM-driven random walk strategy can be used to automatically traverse and explore various government business scenarios, efficiently obtain interface screenshots and scene image information, and thus build a scenario-based image dataset that fits the actual application.

[0037] For the collected images, annotations can typically be made based on the content to be identified, such as the location and content of the content to be identified. In this embodiment of the invention, to improve the accuracy of the sample data, the image feature extraction types can be divided, and then annotations of the feature extraction types can be added to the images. Feature extraction types can include feature recognition types, localization types, inference types, structured output types, etc., and can be specifically determined according to the specific content (image elements) included in the image. Each image can correspond to one or more feature extraction types, that is, each image can correspond to annotation information of one or more feature extraction types.

[0038] Taking government service as an example, the task types can be tailored to the actual needs of government services, adapting multi-dimensional feature extraction types such as identification, positioning, reasoning, and structured output to government-specific applications. For instance, the identification type represents the identification of key elements in government service scenarios, including features with specific formats or shapes such as certificate types, official seals, signature information, form columns, and business items; the positioning type represents the location of core information and physical location in government service scenarios. Core information can include key fields such as name, ID number, unified social credit code, address, amount, and date, while physical location can include locations representing physical addresses such as lobby windows, self-service equipment, specific areas, and problem areas; the reasoning type represents reasoning based on the logic of government business, such as determining whether application materials are complete, whether they meet the processing conditions, whether the materials have been tampered with, and whether they are duplicate submissions, as well as reasoning about the window, department, and required supplementary materials for business processing; the structured output type represents the standardized presentation of government services, specifically including the type of structured output content, such as form field extraction, application material list output, detailed processing procedure summary, approval opinion publicity, and compliance receipt generation.

[0039] Therefore, in this embodiment of the invention, for each image, its included image elements can be identified first, and then it can be determined whether it includes preset key elements, preset core information and physical location, whether it has reasoning requirements, the structured format of the required structured output content, etc., thereby determining the image feature extraction type. Specifically, in this step, the image elements of the image can be classified first to determine the element type, that is, the extraction type to which the element extraction belongs, and then the image feature extraction type can be determined based on the element type. For example, if the image elements include an official seal, ID number, and address, it can be determined that the corresponding element type includes key elements, core information, and physical location, thereby determining that the corresponding feature extraction type includes feature recognition type and positioning type.

[0040] Because the quality of collected images varies, the annotation information generated for each image may also contain errors. Therefore, in this embodiment of the invention, each image and its annotation information in the first image set can be verified to determine the images that pass the verification as image samples, thus obtaining an image sample set. The verification method is not limited; for example, it can verify dimensions such as image clarity, scene completeness, and annotation validity to remove low-quality, noisy, and invalid samples, ensuring the reliability of the basic data. It can also verify whether the annotation information is consistent with the image content, policy clauses, and service guides; whether the annotations of key information such as names, ID numbers, amounts, dates, and document numbers in the images are accurate; and whether each annotation avoids violations such as privacy information leakage, generation of false certificates, and unauthorized interpretation of policies. Specific verification can be implemented using machine sampling, such as OCR accuracy checks, field matching rate verification, and logical compliance rate audits, to ensure the reliability and usability of the annotation data. In some scenarios, manual verification can also be used.

[0041] In another implementation, in this embodiment of the invention, a business corpus can be pre-constructed to generate script templates based on each business corpus. Then, based on these script templates, instructions input during model training are generated for each image. That is, instructions corresponding to each image are pre-generated in the sample set. These instructions are typically input in text format, so in this embodiment, the instructions corresponding to the images can include instruction text. Specifically, the process can be as follows: for each image in the image sample set, the corresponding corpus is matched from the business corpus; the corresponding script template is queried; and instruction text corresponding to the image is generated based on the script template. The business corpus is constructed from voice consultation records, customer service chat records, and comment records associated with preset business types. A first sample set is generated based on the annotation information and instruction text of each image in the image sample set.

[0042] To make the instruction text closer to natural language, we can pre-collect voice consultation records, customer service chat logs, and comment records related to predefined business types to build a business corpus. Then, we can construct script templates for each scenario based on this corpus. For each image, we can determine the corresponding corpus based on its content, then query the corresponding script template, and generate instruction text based on the template, thus ensuring consistency between the image content and the instruction content. For example, if the image content is a social security payment query result image for XX, the corresponding corpus could be a social security payment query scenario. The corresponding script template would be: "(Name) How much social security have you paid?" This would generate the text instruction: "Please extract the amount of social security already paid by XX."

[0043] It should be noted that voice consultation records can include consultation records from government hotlines, recordings of consultations at government service hall windows, etc., while comment records can include real public comments collected from channels such as comment sections of service guides. The business corpus can be archived according to business scenarios, such as social security inquiries, business license processing, household registration transfers, and housing provident fund withdrawals. Based on this real corpus, a script template can be built for each business scenario. The script template can include components such as intent slots, a colloquial variant pool, ellipsis mode switches, and tone modifiers. Then, diverse instruction texts can be generated through slot filling and random combinations.

[0044] It should be noted that in this embodiment of the invention, the structured output format of each image can also be pre-annotated as a tag, such as JSON format tag, two-dimensional table format tag, and entity list format tag. The JSON format can be specifically {"Invoice Code":"110023100","Invoice Number":"008965","Invoice Date":"2025-YY-ZZ","Amount":"129.00"}; the two-dimensional table format can be specifically [["Serial Number","Product Name","Unit Price","Quantity"],["1","Mineral Water","2.00","5"],["2","Bread","5.00","3"]]; the entity list format can be specifically [{"Entity Type":"Price Tag","Price":"19.9","Product":"Milk"},{"Entity Type":"Price Tag","Price":"29.9","Product":"Yogurt"}]. In this embodiment of the invention, after obtaining the image sample set, a first sample set can be generated based on the annotation information, instruction text, and format tags of each image.

[0045] To enhance the robustness of the model in complex real-world scenarios, this embodiment of the invention can also employ multi-dimensional visual noise enhancement strategies for image samples. Specifically, one or more types of noise are randomly superimposed on each image to prevent the model from learning only a single visual feature. Noise types can be categorized into three types: basic noise, which may include Gaussian blur, salt-and-pepper noise, brightness or contrast adjustment, color shift, etc., used to simulate common quality degradation during image acquisition and transmission; scene noise, which may include random text or patterns obscuring local elements, perspective distortion (simulating different shooting angles), and background overlay (superimposing target elements onto a cluttered background, such as placing an invoice on a table or other paper), to enhance the model's adaptability to complex layouts and interfering information; and special noise, which may include handwritten text mixing (simulating handwritten content in forms) and printing blur (simulating low-quality printouts), used to cover common non-ideal situations in government documents, thereby improving the model's ability to analyze diverse materials.

[0046] S102: Obtain a preset second sample set to input into the initial model, obtain the output results, and verify the output results.

[0047] After obtaining the initial model, it can be fine-tuned again based on the second sample set. The implementation method is not limited, such as using RLVR (reinforcement learning based on verifiable rewards).

[0048] The second sample set is pre-generated and may include pre-defined business-related image samples with complex noise and standardized instructions. Inputting these into the initial model yields the output results, i.e., the image feature extraction results. The initial model can then be adjusted by verifying the output results. Standardized instructions represent commands that specify the model's output format. The verification method is not limited and can be implemented using a pre-configured, interactive visual analysis toolset, such as multi-scale cropping tools, OCR refinement tools, format verification tools, compliance verification tools, etc. By using these pre-configured tools to perform multi-dimensional verification of the initial model's output results, the accuracy of extracted field positions, the consistency of extracted characters, and the compliance of output format and business requirements can be verified.

[0049] In another implementation, the second sample set in this step can be selected from high-difficulty samples, such as image samples with high feature extraction difficulty, and noise can be superimposed on them to facilitate the adjustment of the initial model. Specifically, it can be executed as follows: obtain the training data corresponding to the initial model, extract the second image set from the training data where the deviation between the predicted result and the true result is greater than a preset threshold; call the preset noise superposition strategies to superimpose noise on each image in the second image set to obtain the second sample set.

[0050] The initial model training process can include multiple rounds. Each round's training data includes predictions of feature extraction from input image samples. The deviation between the predictions and the actual feature extraction results identifies image samples with poor feature extraction performance, leading to a second image set. Poor feature extraction performance indicates a high difficulty level in feature extraction. The actual feature extraction results are typically labeled on the image samples. The degree of deviation between the predictions and the actual results indicates the number of inconsistent features. A preset threshold represents the minimum threshold for the number of inconsistent features between the predictions and the actual results, such as setting it to 3.

[0051] After obtaining the second image set, noise can be superimposed on each image in the second image set using preset noise superposition strategies. These noise superposition strategies can be pre-defined and may include, for example: nested form noise, where core fields are embedded in multiple layers of tables or sub-forms (e.g., a nested table for "change records" on a business license); multi-entity mixed layout noise, where multiple entities such as certificates, forms, handwritten annotations, and seals are mixed in the same image (e.g., mixing invoices, expense reports, and handwritten signatures); cross-page splicing noise, simulating the situation where scanned documents are truncated across pages and fields are scattered across multiple pages; and semantic ambiguity noise, confusing similar fields (e.g., changing the opening bank to the opening branch, or the issuance date to the effective date).

[0052] It should be noted that, in this embodiment of the invention, the second image set can also be obtained in other ways, such as from special scenarios such as complex environments.

[0053] S103: In response to the verification result not meeting the preset conditions, adjust the initial model based on the verification result and the preset optimization strategy, input the second sample set into the adjusted initial model, obtain the updated output result and perform verification to obtain a new verification result. If the new verification result meets the preset conditions, determine the adjusted initial model as the feature extraction model.

[0054] The validation results represent the validation results across multiple dimensions obtained after validating the initial model output. These may include: whether core fields (such as ID card number, credit code, etc.) contain errors; whether non-core fields (such as remarks, attachment descriptions, etc.) are missing; whether the format conforms to specifications (e.g., date not in AAAA-MM-DD format); and whether the output completely matches the standard answer. The validation results across each dimension can be used to score the output. Preset conditions can be set to be greater than a score threshold. That is, if the score corresponding to the validation result is greater than the score threshold, the validation result is confirmed to meet the preset conditions; if the score corresponding to the validation result is not greater than the score threshold, the validation result is confirmed to not meet the preset conditions. The score threshold can be set according to the scenario; for example, when the scoring range is 0-10, the score threshold can be set to 8 points.

[0055] When the verification results do not meet the preset conditions, the initial model can be adjusted based on the verification results and the preset optimization strategy. The verification results can determine which strategies need to be rewarded and which strategies need to be penalized, thereby adjusting the initial model. The optimization strategy can include reward and penalty strategies, which can be set according to business needs. Table 1 shows one such reward and penalty strategy. In this embodiment of the invention, the PPO (Proximal Policy Optimization) algorithm can be used to score the output results based on the reward strategy and calculate the strategy gradient based on the score, thereby fine-tuning the output strategy of the initial model. For example, the error field extraction logic, format normalization output weights, and tool interaction strategies can be optimized.

[0056] Table 1

[0057] After adjusting the initial model, the second sample set can be input into the adjusted initial model to obtain the updated output result and verify it. Then, based on whether the verification result meets the preset conditions, it is determined whether to adjust the adjusted initial model again. This process is repeated until the new verification result meets the preset conditions or the number of iterations reaches the preset number, and the adjusted initial model, i.e., the feature extraction model, is obtained.

[0058] It should be noted that, in this embodiment of the invention, if the new verification result meets the preset conditions, the adjusted initial model can also be verified using a preset verification sample set. If the verification result meets the requirements (e.g., the average score of the output result corresponding to the verification sample is greater than 9 points and the accuracy of the core field is greater than 99%), then fine-tuning can be stopped.

[0059] In another implementation, the initial model may include a visual feature extraction layer, a policy layer, and an output layer. Through training on the first sample set, the visual feature extraction layer of the initial model usually meets the feature extraction requirements, so it is unnecessary to adjust the visual feature extraction layer in the initial model during this step. Therefore, before performing this step, the model parameters corresponding to the visual feature extraction layer in the initial model can be obtained, and the preset parameters of the model parameters can be updated to the target values. Then, during the initial model adjustment in this step, the following can be done: obtain each model parameter in the initial model, filter out the model parameters whose corresponding preset parameters are not the target values, and adjust the model parameters based on the verification results and the preset optimization policy. By freezing the visual feature extraction layer, only the output layer and the policy layer are optimized, avoiding damage to the already learned basic structural capabilities.

[0060] Model parameters represent the parameters to be adjusted during model training, while preset parameters represent whether the model parameters should be adjusted. Setting the value of the preset parameter corresponding to the model parameter to the target value means that the model parameter will not be adjusted during training. Therefore, in this step, the preset parameters of the model parameters corresponding to the visual feature extraction layer can be updated to the target value in advance, so that when adjusting the initial model, the model parameters to be adjusted can be selected according to the target value.

[0061] S104: In response to the image feature extraction instruction, acquire the image to be processed, input it into the feature extraction model, and obtain structured image features.

[0062] After completing the model training, a feature extraction model can be obtained, which can be used to extract features from images in the preset business. That is, after receiving the image feature extraction instruction, the image to be processed is obtained and input into the feature extraction model to obtain structured image features.

[0063] In this embodiment of the invention, the basic large model is trained in two stages. In the first stage, through a pre-labeled sample set, the initial model can quickly grasp the visual structure, semantic association, and basic structured output capability of the image, making the initial model more adaptable. In the second stage, the initial model is adjusted according to the optimization strategy, so that the feature extraction model has higher feature extraction accuracy, that is, the feature extraction model has higher adaptability and accuracy. In this way, the trained feature extraction model processes the structured image features of the image to be processed, resulting in higher accuracy of the extraction results.

[0064] The following is combined with Figure 1 The illustrated embodiment provides a detailed explanation of the model training process in the image information processing method of this invention. Figure 2 The following is a detailed flowchart of the process.

[0065] like Figure 2 As shown, the multimodal large model represents the basic large model; the SFT data preparation stage can represent the collection of images related to a preset business, obtaining a first image set, determining the annotation information, instruction text, and structured output format of the images in the first image set, as well as the verification process, and then generating a first sample set, i.e., the SFT fine-tuning dataset; the multimodal large model can be trained in the first stage using the SFT fine-tuning dataset, i.e., fine-tuning the multimodal large model, to obtain the fine-tuned multimodal large model; RLVR data preparation represents the construction of a second image set; after processing the second image set with noise superposition, a second sample set can be obtained, i.e., the RLVR fine-tuning dataset; then, the fine-tuned multimodal large model can be further enhanced and fine-tuned using the RLVR fine-tuning dataset to obtain the enhanced and fine-tuned multimodal large model, i.e., the feature extraction model.

[0066] In this embodiment of the invention, a two-stage (SFT stage and RLVR stage) fine-tuning mode is adopted to achieve hierarchical iteration and precise optimization of the model's structured output capability. At the same time, a dedicated dataset adapted to the two-stage training is constructed to provide reliable support for the fine-tuning process.

[0067] The supervised fine-tuning stage enables comprehensive scene coverage, ensuring data coverage of various typical application scenarios, visual environments, and content domains, giving the model broad scene generalization capabilities; it achieves comprehensive feature extraction type coverage, encompassing multiple task forms such as recognition, localization, reasoning, and structured output, guiding the model to learn multi-dimensional image structures and semantic understanding methods; it achieves high-quality annotation alignment, ensuring high consistency between instruction text, semantic description, and image content, providing the model with accurate and reliable supervision signals; and it achieves unified instruction format specifications, improving the model's ability to understand and follow human intentions.

[0068] Strengthening the fine-tuning stage can improve the accuracy of structured information extraction in complex scenarios (high noise, multi-element nesting, cross-modal ambiguity), reduce the rate of missing and incorrect field extraction; enhance the model's adherence to the standardized output format of preset business scenarios, and eliminate problems such as format chaos and missing fields; enable the model to have the ability of "self-verification and correction", and actively optimize the extraction results through interaction with visual analysis tools and annotations, so as to adapt to the compliance and rigor requirements of government services.

[0069] It should be noted that the data processing principle in the embodiments of the present invention is the same as... Figure 1 The data processing principles in the illustrated embodiments are the same and will not be repeated here.

[0070] To address the problems existing in the prior art, embodiments of the present invention provide an image information processing apparatus 300, such as... Figure 3 As shown, the device 300 includes: a training unit 301, used to acquire a preset basic large model and a first sample set, call a first training algorithm, train the basic large model based on the first sample set, and obtain an initial model, wherein the first sample set includes pre-labeled image samples; The training unit 301 is also used to obtain a preset second sample set, input it into the initial model, obtain the output result, and verify the output result; The adjustment unit 302 is used to adjust the initial model based on the verification result and the preset optimization strategy in response to the verification result not meeting the preset conditions. The second sample set is input into the adjusted initial model to obtain an updated output result and perform verification to obtain a new verification result. If the new verification result meets the preset conditions, the adjusted initial model is determined as the feature extraction model. The extraction unit 303 is used to respond to the image feature extraction instruction, acquire the image to be processed, input it into the feature extraction model, and output structured image features.

[0071] It should be understood that the manner in which embodiments of the present invention are implemented is different from the implementation method. Figure 1 The embodiments shown are the same and will not be described again here.

[0072] In one embodiment, the device 300 further includes: The acquisition unit is used to acquire a first set of images associated with a preset business type; The generation unit is used to identify the image elements of each image in the first image set, determine the feature extraction type of the image based on the image elements, and generate annotation information corresponding to the feature extraction type. The generation unit is further configured to verify each image in the first image set and the annotation information of each image, and determine an image sample set from the first image set based on the verification result, so as to generate the first sample set.

[0073] In yet another embodiment, the generating unit is specifically used for: Identify the image elements of the image, classify the image elements, and determine the types of elements included in the image; The feature extraction type of the image is determined based on the element type.

[0074] In yet another embodiment, the generating unit is specifically used for: For each image in the image sample set, the corresponding corpus is matched from the business corpus, the corresponding script template is queried from the corpus, and the instruction text corresponding to the image is generated based on the script template. The business corpus is constructed from voice consultation records, customer service chat records, and comment records associated with preset business types. The first sample set is generated based on the annotation information and instruction text of each image in the image sample set.

[0075] In yet another embodiment, the extraction unit 303 is further configured to: Obtain the training data corresponding to the initial model, and extract a second set of images from the training data whose deviation between the predicted result and the actual result is greater than a preset threshold. The preset noise overlay strategies are invoked to overlay noise onto each image in the second image set, resulting in the second sample set.

[0076] In yet another embodiment, the initial model includes a visual feature extraction layer; The device 300 further includes: The update unit is used to obtain the model parameters corresponding to the visual feature extraction layer in the initial model and update the preset parameters of the model parameters to the target value. The adjustment unit 302 is specifically used for: Obtain the model parameters in the initial model, filter the model parameters to be adjusted that are not the target values ​​of the preset parameters, and adjust the model parameters to be adjusted based on the verification results and the preset optimization strategy.

[0077] It should be understood that the manner in which embodiments of the present invention are implemented is different from the implementation method. Figure 1 , 2 The embodiments shown are the same and will not be described again here.

[0078] In this embodiment of the invention, the basic large model is trained in two stages. In the first stage, through a pre-labeled sample set, the initial model can quickly grasp the visual structure, semantic association, and basic structured output capability of the image, making the initial model more adaptable. In the second stage, the initial model is adjusted according to the optimization strategy, so that the feature extraction model has higher feature extraction accuracy, that is, the feature extraction model has higher adaptability and accuracy. In this way, the trained feature extraction model processes the structured image features of the image to be processed, resulting in higher accuracy of the extraction results.

[0079] According to embodiments of the present invention, an electronic device and a readable storage medium are also provided.

[0080] An electronic device according to an embodiment of the present invention includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the image information processing method provided in the embodiment of the present invention.

[0081] Figure 4 An exemplary system architecture 400 is shown, in which the image information processing method or apparatus of the present invention can be applied.

[0082] like Figure 4 As shown, system architecture 400 may include terminal devices 401, 402, and 403, a network 404, and a server 405. Network 404 serves as the medium for providing communication links between terminal devices 401, 402, and 403 and server 405. Network 404 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0083] Users can use terminal devices 401, 402, and 403 to interact with server 405 via network 404 to receive or send messages, etc. Various client applications can be installed on terminal devices 401, 402, and 403.

[0084] Terminal devices 401, 402, and 403 can be various electronic devices with displays that support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0085] Server 405 can be a server that provides various services. The server can analyze and process data such as received product information query requests, and feed back the processing results (such as product information - just an example) to the terminal device.

[0086] It should be noted that the image information processing method provided in the embodiments of the present invention is generally executed by server 405, and correspondingly, the image information processing device is generally set in server 405.

[0087] It should be understood that Figure 4 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0088] The following is for reference. Figure 5 It shows a schematic diagram of the structure of a computer system 500 suitable for implementing embodiments of the present invention. Figure 5 The computer system shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0089] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 502 or programs loaded from storage section 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the system 500. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0090] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 510 as needed so that computer programs read from it can be installed into storage section 508 as needed.

[0091] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit (CPU) 501, it performs the functions defined above in the system of this invention.

[0092] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0093] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a unit, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0094] The units described in the embodiments of the present invention can be implemented in software or hardware. The described units can also be located in a processor; for example, a processor can be described as including a training unit, an adjustment unit, and an extraction unit. The names of these units do not necessarily limit the specific unit; for example, a training unit can also be described as a "unit for training functions."

[0095] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to perform the image information processing method provided by the present invention.

[0096] In another aspect, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the image information processing method provided in the embodiments of the present invention.

[0097] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for image information processing, characterized in that, include: Obtain a preset basic large model and a first sample set, call the first training algorithm, train the basic large model based on the first sample set, and obtain an initial model. The first sample set includes pre-labeled image samples. Obtain a preset second sample set, input it into the initial model, obtain the output result, and verify the output result; In response to the verification result not meeting the preset conditions, the initial model is adjusted based on the verification result and the preset optimization strategy. The second sample set is input into the adjusted initial model to obtain an updated output result, which is then verified to obtain a new verification result. If the new verification result meets the preset conditions, the adjusted initial model is determined as the feature extraction model. In response to an image feature extraction instruction, an image to be processed is acquired and input into the feature extraction model to obtain structured image features.

2. The method according to claim 1, characterized in that, Before obtaining the pre-defined basic large model and the first sample set, the following steps are included: Collect the first set of images associated with the preset business type; For each image in the first image set, identify the image elements of the image, determine the feature extraction type of the image based on the image elements, and generate annotation information corresponding to the feature extraction type; The annotation information of each image in the first image set is verified, and an image sample set is determined from the first image set based on the verification results to generate the first sample set.

3. The method according to claim 2, characterized in that, Identifying image elements of the image and determining the feature extraction type of the image based on the image elements, including: Identify the image elements of the image, classify the image elements, and determine the types of elements included in the image; The feature extraction type of the image is determined based on the element type.

4. The method according to claim 2, characterized in that, Generating the first sample set includes: For each image in the image sample set, the corresponding corpus is matched from the business corpus, the corresponding script template is queried from the corpus, and the corresponding instruction text is generated based on the script template. The business corpus is constructed from voice consultation records, customer service chat records and comment records associated with preset business types. The first sample set is generated based on the annotation information and instruction text of each image in the image sample set.

5. The method according to claim 1, characterized in that, Before obtaining the preset second sample set, the following steps are included: Obtain the training data corresponding to the initial model, and extract a second set of images from the training data whose deviation between the predicted result and the actual result is greater than a preset threshold. The preset noise overlay strategies are invoked to overlay noise onto each image in the second image set, resulting in the second sample set.

6. The method according to claim 1, characterized in that, The initial model includes a visual feature extraction layer; Before adjusting the initial model based on the verification results and the preset optimization strategy, the process also includes: Obtain the model parameters corresponding to the visual feature extraction layer in the initial model, and update the preset parameters of the model parameters to the target values; Adjusting the initial model based on the verification results and a preset optimization strategy includes: Obtain the model parameters in the initial model, filter the model parameters to be adjusted that are not the target values ​​of the preset parameters, and adjust the model parameters to be adjusted based on the verification results and the preset optimization strategy.

7. An apparatus for image information processing, characterized in that, include: The training unit is used to obtain a preset basic large model and a first sample set, call a first training algorithm, train the basic large model based on the first sample set, and obtain an initial model. The first sample set includes pre-labeled image samples. The training unit is also used to obtain a preset second sample set, input it into the initial model, obtain the output result, and verify the output result; An adjustment unit is used to adjust the initial model based on the verification result and a preset optimization strategy in response to the verification result not meeting the preset conditions. The second sample set is input into the adjusted initial model to obtain an updated output result and perform verification to obtain a new verification result. If the new verification result meets the preset conditions, the adjusted initial model is determined as the feature extraction model. The extraction unit is used to respond to the image feature extraction instruction, acquire the image to be processed, input it into the feature extraction model, and output structured image features.

8. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

Images