Image recognition model construction method, device, equipment and medium

Abstract

The present disclosure relates to a method and device for constructing an image recognition model, and a medium. The method comprises constructing an initial model and iteratively training the initial model. The training process of any training cycle comprises iteratively training a first training model of the current training cycle using labeled samples of the previous training cycle until a second training model is obtained. Before the iterative training of the first training model is completed, the first training model is used to label and identify the current unlabeled samples to obtain a labeling and identifying result. An artificial review result of the labeling and identifying result is obtained, and based on the artificial review result, the labeled samples of the current training cycle are determined. By applying the technical solution of the present disclosure, the model can be trained while being labeled, the training of the model does not require additional time, the model training can be completed as soon as the labeling is completed, the model construction period is greatly shortened, and the model construction efficiency can be effectively improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and medium for constructing an image recognition model. Background Technology

[0002] With the continuous development of technology, automated intelligent technology and artificial intelligence technology have brought many conveniences to people's lives. In the process of information acquisition, language and text are the most basic and important means for us to obtain information. In the field of image recognition technology, there is another very important subfield—Optical Character Recognition (OCR). This refers to the process of using optical equipment to examine characters printed on paper, determining their shape by detecting dark and light patterns, and then translating the image into computer text using character recognition methods—that is, computer reading of text.

[0003] Currently, the traditional method of building non-fixed OCR models is based on manual annotation. All sample images need to be manually annotated before model training can be carried out and a recognition model can be built.

[0004] However, traditional model building methods have long cycles, ranging from three months to a year before they can be deployed for business use, often failing to keep up with business development needs. Furthermore, the large amount of annotation required and the low timeliness of model performance verification mean that key stages of model building require frequent participation from business personnel, such as the annotation stage before model training and the performance verification stage after training, thus increasing labor costs. Summary of the Invention

[0005] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, this disclosure provides a method, apparatus, device and medium for constructing an image recognition model.

[0006] Firstly, this disclosure provides a method for constructing an image recognition model, including:

[0007] Construct an initial model and iteratively train the initial model; wherein the training process in any training cycle includes:

[0008] The labeled samples from the previous training cycle are used to iteratively train the first training model in the current training cycle until the second training model is obtained.

[0009] Before the first training model completes its iterative training, the first training model is used to label and identify the currently unlabeled samples to obtain the labeling and identification results.

[0010] Obtain the manual review results of the labeled recognition results, and determine the labeled samples for the current training period based on the manual review results.

[0011] Optionally, in any of the training cycles, the method further includes:

[0012] Obtain the recognition accuracy of the labeled target samples;

[0013] Based on the recognition accuracy of the labeled samples of the target, the model labeling effect of the first training model is determined;

[0014] Based on the model annotation results, the target annotation mode is determined, wherein the target annotation mode includes: manual annotation mode and model annotation mode;

[0015] When the target annotation mode is manual annotation mode, the annotation data of the user for the unannotated samples is obtained to obtain the labeled samples.

[0016] Optionally, obtaining the recognition accuracy of the labeled target samples includes:

[0017] Get the recognition accuracy of the labeled samples in the current training period; or,

[0018] Obtain the recognition accuracy of all labeled samples; or,

[0019] Obtain the recognition accuracy of all labeled samples excluding those manually labeled.

[0020] Optionally, determining the target annotation pattern based on the model annotation effect includes:

[0021] When the model annotation effect does not meet the preset conditions, the target annotation mode is determined to be the manual annotation mode; or...

[0022] When the model annotation effect does not meet the preset conditions, a prompt message is output to ask the user whether to switch the annotation mode; and the target annotation mode is determined based on the user's operation information in response to the prompt message.

[0023] Optionally, in any of the training cycles, the method further includes:

[0024] When the current annotation mode is manual annotation mode, determine whether the preset switching conditions are met;

[0025] When the switching conditions are met, the manual annotation mode is switched to the model annotation mode.

[0026] Optionally, determining whether a preset switching condition is met includes:

[0027] The switching condition is determined to be met when the number of manually labeled items reaches a preset number, or when a user's mode switching instruction is received, or when a new training model is obtained through training the results of manually labeled items.

[0028] Optionally, the construction of the initial model includes:

[0029] Obtain a labeling template, which describes the labeling content to be labeled and recognized by the image recognition model;

[0030] Based on the annotation template, the user is prompted to manually annotate a specified number of first image samples and obtain second image samples containing manually annotated data; wherein, the specified number satisfies the minimum sample requirement for training the initial model;

[0031] The initial model is obtained by training the model using the second image sample.

[0032] Optionally, the annotation template includes: a fixed format template and a non-fixed format template;

[0033] The fixed template is used to describe how the image recognition model identifies the labeled content according to fixed labeled positions.

[0034] The non-fixed template is used to describe how the image recognition model annotates and recognizes the labeled content at non-fixed positions in the image.

[0035] Optionally, in any of the training cycles, the method further includes:

[0036] The second training model trained in the current training cycle is put online and used to label and identify unlabeled samples in the next training cycle, thus obtaining the labeling and identification results.

[0037] Optionally, the type of the sample includes at least one of the following:

[0038] Images of identification documents, receipts, and invoices.

[0039] Secondly, this disclosure provides an apparatus for constructing an image recognition model, comprising:

[0040] The building module is configured to build the initial model;

[0041] The training module is configured to iteratively train the initial model; wherein the training process of any training cycle includes: iteratively training the first training model of the current training cycle using the labeled samples of the previous training cycle until a second training model is obtained; before the first training model completes its iterative training, using the first training model to label and identify the currently unlabeled samples to obtain the labeling and identification results; obtaining the manual review results of the labeling and identification results, and determining the labeled samples of the current training cycle based on the manual review results.

[0042] Thirdly, this disclosure provides an electronic device, including:

[0043] processor;

[0044] Memory, used to store executable instructions;

[0045] The processor is used to read executable instructions from memory and execute the executable instructions to implement the image recognition model construction method described in the first aspect.

[0046] Fourthly, this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the image recognition model construction method described in the first aspect.

[0047] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0048] Compared with traditional model building methods, the image recognition model construction method, apparatus, device, and medium disclosed herein provide a technical solution that, during the image recognition model construction process, first constructs an initial model, and then iteratively trains the initial model. The training process for any training cycle includes: iteratively training the first training model for the current training cycle using labeled samples from the previous training cycle until a second training model is obtained; before the first training model completes its iterative training, labeling and recognizing the currently unlabeled samples using the first training model to obtain the labeling and recognition results; then obtaining the manual review results of the labeling and recognition results, and determining the labeled samples for the current training cycle based on the manual review results. The technical solution provided herein can simultaneously label and recognize currently unlabeled samples and iteratively train the training model for the current training cycle using labeled samples from the previous training cycle, thus achieving the effect of simultaneous labeling and training of the model. Model training does not consume additional time; model training is completed once labeling is finished, greatly shortening the model construction cycle and effectively improving model construction efficiency, thereby improving the efficiency of image recognition. By applying the technical solution disclosed herein, the workload of manual annotation can be greatly reduced, and the user threshold can be lowered. Without relying on data scientists for modeling, business personnel without technical backgrounds can also annotate to complete the construction and use of image recognition models, saving the human resource costs of business personnel participating in model construction. Attached Figure Description

[0049] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0050] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 A flowchart illustrating a method for constructing an image recognition model according to an embodiment of this disclosure is shown.

[0052] Figure 2 A schematic diagram of the system architecture of an application example provided in this disclosure is shown;

[0053] Figure 3 A flowchart illustrating an example of constructing an initial model provided in an embodiment of this disclosure is shown;

[0054] Figure 4A flowchart illustrating the intelligent switching process of annotation modes provided in an embodiment of this disclosure is shown.

[0055] Figure 5 A schematic diagram of the structure of an image recognition model construction apparatus provided in an embodiment of this disclosure is shown;

[0056] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0057] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0058] While some embodiments of this disclosure are shown in the accompanying drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0059] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0060] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0061] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0062] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0063] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0064] Figure 1 The diagram illustrates a flowchart of a method for constructing an image recognition model provided in this embodiment.

[0065] In this embodiment, the image recognition model can be an OCR-based recognition model or other image recognition models. The method for constructing this image recognition model can be executed by an electronic device. The electronic device can include, but is not limited to, smart terminals such as computers, laptops, digital radio receivers, personal digital assistants (PDAs), tablet computers (PADs), portable media players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable devices, and fixed terminals such as digital TVs, desktop computers, and smart home devices.

[0066] It should be noted that the image recognition model described in this embodiment of the invention can be used to predict image categories, text categories, etc. The image recognition model aims to predict issues related to objects or events in relevant scenarios. For example, it can be used to predict image categories, predict text within images, predict the text category within images, etc., so that the prediction results can be directly used as a decision-making basis or further combined with other rules to become a decision-making basis. Furthermore, the image recognition model described in this embodiment of the invention can be used in scenarios such as privacy computing, multi-party secure computing, federated learning, anonymous queries, and secure (privacy) intersection.

[0067] Furthermore, the image recognition model in the embodiments of the present invention can be used in scenarios including but not limited to the following:

[0068] Image processing scenarios include: Optical Character Recognition (OCR), face recognition, object recognition, and image classification. More specifically, OCR can be applied to document (such as invoice) recognition, handwritten character recognition, etc.; face recognition can be applied to security and other fields; object recognition can be applied to traffic sign recognition in autonomous driving scenarios; and image classification can be applied to "photo shopping" and "finding similar items" on e-commerce platforms.

[0069] Natural language processing scenarios include: text review (such as contracts, legal documents, and customer service records), spam content identification (such as spam SMS identification), and text classification (sentiment, intent, and topic).

[0070] Intelligent question-and-answer scenarios include: chatbots and intelligent customer service;

[0071] Business decision-making scenarios include those in the fintech, healthcare, and municipal sectors, among which:

[0072] The fintech sector includes: marketing (such as coupon usage prediction, ad click behavior prediction, user profiling, etc.) and customer acquisition, anti-fraud, anti-money laundering, underwriting and credit scoring, and commodity price prediction;

[0073] The medical field includes: disease screening and prevention, personalized health management, and assisted diagnosis;

[0074] The municipal sector includes: social governance and regulatory enforcement, resource, environmental and facility management, industrial development and economic analysis, public services and livelihood security, and smart cities (allocation and management of various urban resources such as public transportation, ride-hailing services, and shared bicycles).

[0075] Recommended business scenarios include: recommendations for news, advertisements, music, information, videos, and financial products (such as wealth management and insurance);

[0076] Search scenarios include: web search, image search, text search, video search, etc.

[0077] Abnormal behavior detection scenarios include: detection of abnormal electricity consumption behavior of State Grid customers, detection of malicious network traffic, and detection of abnormal behavior in operation logs.

[0078] like Figure 1 As shown, the construction process of this image recognition model includes the following steps.

[0079] S110, Construct the initial model.

[0080] The initial model can be an initial model for image recognition, image detection, or image structuring. Subsequent iterations of training samples (annotated sample images) are needed to train the initial model, ultimately yielding the image recognition model. This image recognition model can be used to identify annotated content in a target image (the image to be recognized); that is, the input to the image recognition model is the image to be recognized, and the output can be the annotated content. In some embodiments, the output may further include the annotation location corresponding to the annotated content. The sample images serve as the basic data for training the image recognition model. Optionally, the sample types may include, but are not limited to, at least one of the following: document images, receipt images, invoice images, etc.

[0081] In this embodiment, when constructing the initial model, manually labeled data based on sample images is required, including user annotations on the sample images. The amount of manually labeled data needs to meet the minimum sample requirement for training the initial model. In this embodiment, the minimum sample requirement for training the initial model refers to the minimum number of samples needed to initially train a model (even if the model's labeling performance is not up to standard). For example, if only 10 samples are needed to initially train a model, then the minimum sample requirement for training the initial model is 10 images. Therefore, when constructing the initial model, only 10 manually labeled sample data images need to be manually labeled by the user to initially train an initial model. It should be understood that the manually labeled samples used to construct the initial model can be a small number, far less than the number of samples required for training an image recognition model. For example, training a well-labeled image recognition model may require hundreds of samples, but when constructing the initial model, only 10 user-labeled images may be needed, far less than the number of samples required to train a mature model.

[0082] S120. Iteratively train the initial model.

[0083] This embodiment utilizes machine learning algorithms for model training. The specific model training process can be executed on the electronic device side or sent to a server for execution. The resulting image recognition model can be used for image recognition; for example, a recognition model trained on a document sample image can be used to recognize document content within a document image, while a recognition model trained on a bill sample image can be used to recognize bill content within a bill image.

[0084] This embodiment divides the initial model training process into multiple training cycles until an image recognition model is obtained through iterative training. A single training cycle consists of the period from the start of training the first training model to the completion of training the second training model. Each training cycle contains its own corresponding unlabeled samples, which can be real-time data samples. The training process for any given training cycle includes:

[0085] (1) Use the labeled samples from the previous training cycle to iteratively train the first training model in the current training cycle until the second training model is obtained.

[0086] (2) Before the first training model is completed in iteration, the first training model is used to label and identify the currently unlabeled samples to obtain the labeling and identification results.

[0087] (3) Obtain the manual review results of the labeling and recognition results, and determine the labeled samples for the current training cycle based on the manual review results.

[0088] In this context, the execution order of (1) and (2) can be parallel. The annotation recognition result is obtained by recognizing the unannotated sample 1 in the current training cycle, while the annotation data of the labeled sample is the annotation data obtained for the unannotated sample 2 in the previous training cycle. The manual review result is obtained by further manual review of the annotation recognition result. It should be understood that if the annotation recognition result is correct, no modification is needed during manual review, and the manual review result is the same as the annotation recognition result. Then, based on the annotation recognition result and the unannotated sample, the labeled sample in the current training cycle can be obtained. Alternatively, if there is a problem with the annotation recognition result, it needs to be corrected with manual review. Therefore, the manual review result is different from the annotation recognition result. Then, based on the manual review result and the unannotated sample, the labeled sample in the current training cycle can be obtained.

[0089] For example, taking three adjacent training periods A, B, and C as an example, after the model has been trained in training period A, during the training process of training period B, the labeled sample a of training period A is used to iteratively train the training model B1 of training period B (which is the same as the training model A2 obtained by training model A1 in training period A) until the training model B2 is obtained. This training model B2 can be used as the training model C1 of the subsequent training period C for subsequent iterative training.

[0090] Before the training model B1 completes its iterative training, the unlabeled sample b is labeled and identified using the training model B1 to obtain the labeling and identification results. For example, the image of the unlabeled sample b is input into the training model B1, and the labeling and identification results of sample b are output. The labeling and identification results of sample b can be manually reviewed. The labeler can choose to modify or confirm the content of the model labeling. Finally, the manual review results of the labeling and identification results of sample b are obtained, and based on the manual review results, the labeled samples b of training period B are determined. These labeled samples b can be used to iteratively train the training model C1 (which is the same as the training model B2) of training period C until the training model C2 is obtained.

[0091] It should be noted that the solution provided in this embodiment can cover printed text scenarios, handwritten text scenarios, mixed printed and handwritten text scenarios, English text scenarios, etc., and has strong scalability.

[0092] Compared to traditional model building methods, the image recognition model construction method provided in this embodiment can label and recognize currently unlabeled samples within each training cycle, and iteratively train the model in the current training cycle using labeled samples from the previous training cycle. This achieves the effect of labeling and training the model simultaneously, without additional time consumption for model training. Model training is completed as soon as labeling is finished, greatly shortening the model building cycle and effectively improving model building efficiency, thereby improving image recognition efficiency. By applying the technical solution of this embodiment, the user threshold is lowered. It does not rely on data scientists for modeling; business personnel without technical backgrounds can also label and complete the construction and use of image recognition models, saving the manpower costs of business personnel involved in model building.

[0093] Optionally, in any training cycle, the method of this embodiment may further include: deploying the second training model trained in the current training cycle online for labeling and recognizing unlabeled samples in the next training cycle, obtaining the labeled and recognized result, which is the first training model for the next training cycle. This embodiment uses this optional method to achieve the purpose of iteratively updating the training model, thereby obtaining an image recognition model with high recognition accuracy.

[0094] Furthermore, as a refinement and extension of the above embodiments, optionally, in any training cycle, the method of this embodiment may further include: firstly, obtaining the recognition accuracy of the target labeled samples; then, based on the recognition accuracy of the target labeled samples, determining the model labeling effect of the first training model; then, based on the model labeling effect, determining the target labeling mode, wherein the target labeling mode may include: manual labeling mode and model labeling mode; when the target labeling mode is manual labeling mode, obtaining the user's labeling data for unlabeled samples to obtain labeled samples.

[0095] The manual annotation mode can be a mode in which the image samples to be annotated need to be manually annotated; the model annotation mode can be a mode in which the model automatically provides machine annotations for the image samples to be annotated, and the process shown in (2) to (3) in step S120 above can be referred to. Accordingly, the recognition accuracy of the target annotated sample can be the degree of matching between the annotation recognition result of the training model in the current training cycle and the manual review result.

[0096] For example, obtaining the recognition accuracy of the target labeled samples may specifically include: obtaining the recognition accuracy of the labeled samples in the current training period; or, obtaining the recognition accuracy of all labeled samples; or, obtaining the recognition accuracy of all labeled samples excluding those manually labeled.

[0097] For example, regarding the annotation and recognition results of the machine-annotated samples for the training model in the current training period n, the annotator can choose to modify or confirm these machine-annotated contents. Then, based on the annotation modifications, the recognition accuracy of the labeled samples in the current training period n can be calculated. In this way, based on the real-time recognition accuracy of the labeled samples, it can be determined in a timely manner whether to switch back to manual annotation mode, thus achieving timely switching of annotation modes, reducing the amount of modification required by users, and improving annotation efficiency.

[0098] For example, if the current training period is training period n, and n training periods have already been completed, the recognition accuracy of all labeled samples is calculated based on the machine annotation modifications made by the annotators to all labeled samples during these n training periods. In this way, the overall recognition accuracy of labeled samples across multiple training periods can be used to accurately determine whether to switch back to manual annotation mode, thus achieving precise switching of annotation modes.

[0099] For example, if the current training period is training period n, and n training periods have already been completed, the recognition accuracy of all labeled samples (excluding manually labeled samples) can be calculated based on the machine annotation modifications made by the annotators to all labeled samples (excluding manually labeled samples) during these n training periods. In this way, based on the overall recognition accuracy of labeled samples (excluding manually labeled samples) across multiple training periods, it is possible to more accurately determine whether to switch back to manual annotation mode, thus enabling precise control over the timing of annotation mode switching.

[0100] This embodiment determines the model annotation effect of the training model based on the recognition accuracy of the target labeled samples. This effect can be specifically represented by a numerical value, such as the average recognition accuracy. In this embodiment, the current annotation mode (manual annotation mode or model annotation mode) can be determined based on the model annotation effect of the training model. For example, determining the target annotation mode based on the model annotation effect may include: determining the target annotation mode as manual annotation mode when the model annotation effect does not meet preset conditions; or, when the model annotation effect does not meet preset conditions, outputting a prompt message to the user to switch the annotation mode; and determining the target annotation mode based on the user's operation information in response to the prompt message.

[0101] For example, based on the average recognition accuracy of the labeled samples, it can be determined whether the model's annotation performance meets preset conditions. If the average recognition accuracy is greater than or equal to a certain threshold (e.g., 50%), it indicates that the currently trained model has a high recognition accuracy, and the model's annotation performance meets the preset conditions. The model annotation mode can continue to be used to reduce the workload of manual annotation, thus eliminating the need to switch the annotation mode. Conversely, if the average recognition accuracy is less than a certain threshold (e.g., 50%), it indicates that the currently trained model has a low recognition accuracy, and the model's annotation performance does not meet the preset conditions. Continuing to use the model annotation mode would increase the workload of manual annotation modification, thus requiring a switch to manual annotation mode. In practical applications, the annotation mode can be automatically switched to manual annotation mode when the model annotation performance does not meet the preset conditions, or a prompt message can be output allowing the user to choose whether to switch annotation modes. Based on the user's selection, the appropriate annotation mode can be determined: switch to manual annotation mode or continue using model annotation mode.

[0102] After switching to manual annotation mode, more effective user-annotated data is accumulated to train a model with higher image recognition accuracy, thereby helping users perform machine annotation and saving the workload of manual annotation. Therefore, optionally, in any training cycle, the method of this embodiment may further include: when the current annotation mode is manual annotation mode, determining whether a preset switching condition is met; when the switching condition is met, switching from manual annotation mode to model annotation mode.

[0103] For example, the above determination of whether the preset switching conditions are met may specifically include: determining that the switching conditions are met when the number of manually labeled data reaches a preset number, or when a user's mode switching instruction is received, or when a new training model is obtained through training the results of manually labeled data.

[0104] For example, in manual annotation mode, if the number of manually annotated samples reaches a preset number or even higher, the switching conditions are met, and the mode is switched to model annotation mode. Alternatively, the user can choose whether to switch modes; if a switch is required based on the user's instruction, the switching conditions are met, and the mode is switched to model annotation mode. Or, after accumulating more effective user-annotated data, a model with higher image recognition accuracy can be trained, at which point the switching conditions are met, and the mode is switched to model annotation mode.

[0105] After switching back to model annotation mode, in subsequent training cycles, it can be determined whether to switch back to manual annotation mode based on the model's accuracy in recognizing samples. The specific judgment process described above will not be repeated here. This embodiment enables intelligent switching of annotation modes to quickly switch to a mode more suitable for improving annotation efficiency. It dynamically provides the most user-friendly annotation experience, thereby improving annotation efficiency and allowing users to promptly understand the recognition accuracy of the image recognition model trained in real time.

[0106] It should be noted that, in this embodiment, if a switching of annotation mode (from model annotation mode to manual annotation mode, or vice versa) is determined during any training cycle, the switching can be performed in real-time within the current training cycle or in the next training cycle. This embodiment, by switching the annotation mode in real-time within the current training cycle, allows for timely correction, enabling a quick switch back to manual annotation mode when automatic model recognition is no longer suitable, thus reducing the amount of user modifications and improving annotation efficiency and accuracy. Switching the annotation mode in the next training cycle, while ensuring timely switching to a suitable annotation mode, improves the smoothness of the entire annotation process and reduces the impact of real-time annotation mode changes on user operations.

[0107] Furthermore, as a refinement and extension of the above embodiments, to illustrate the process of constructing the initial model, step S110 may optionally include: firstly, obtaining a labeling template, which can be used to describe the labeling content to be labeled and recognized by the image recognition model; then, based on the labeling template, prompting the user to manually label a specified number of first image samples, and obtaining second image samples containing manually labeled data; wherein, the specified number satisfies the minimum sample requirement for training the initial model; and then using the second image samples containing manually labeled data to train the model, thereby obtaining the initial model.

[0108] In this embodiment, the annotation template can be user-created, modified from an existing template, or selected from an existing template library. These options allow users to freely choose the annotation template based on their specific needs, thus satisfying a wider range of practical requirements.

[0109] For example, the annotation template may include a fixed-format template and a non-fixed-format template. The fixed-format template can be used to describe how the image recognition model annotates and identifies content at fixed locations; while the non-fixed-format template can be used to describe how the image recognition model annotates and identifies content at non-fixed locations within the image. This embodiment uses a non-fixed-format template to assist in the accurate construction of the image recognition model, enabling subsequent accurate recognition of non-fixed-format invoices, documents, certificates, etc., demonstrating good versatility.

[0110] In this embodiment, the specified number can be related to the annotation template. Depending on the amount of content to be annotated in the annotation template, the required specified number can be different. It should be noted that in this embodiment, the specified number that meets the minimum requirement means that it can be equal to or greater than the minimum requirement. In actual scenarios, the specified number is much smaller than the total number of samples (including all samples used to train the image recognition model).

[0111] When prompting users to manually annotate a specified number of image samples based on an annotation template, the system can guide users on which content to annotate in the image samples according to the annotation content to be recognized by the image recognition model described in the annotation template. In addition, the system can also prompt users for a corresponding number of image samples to manually annotate based on the type of annotation template (such as a fixed-format template or a non-fixed-format template). For example, for a fixed-format template, the system can prompt users to manually annotate one image sample, while for a non-fixed-format template, the system can prompt users to manually annotate ten image samples.

[0112] It should be noted that the prompt indicating the specified number can be explicit to the user, such as reminding the user to annotate 10 image samples in the form of text or voice; the prompt can also be implicit to the user, such as directly switching to automatic model annotation mode after the user has annotated 10 image samples, at which time the user can be reminded to enter the model annotation mode, but this reminder can be unrelated to the specified number.

[0113] To illustrate the specific implementation process of the above embodiments, the construction process of an OCR recognition model is taken as an example, and the following application examples are given, but not limited to these:

[0114] Currently, the construction of traditional OCR recognition models still heavily relies on manual annotation. All sample images need to be manually annotated before model training can begin, resulting in a recognition model. This leads to long model construction cycles, a large amount of annotation work, and low timeliness in model effectiveness verification. Key stages of model construction require frequent participation from business personnel. However, due to the tedious and large volume of annotation work and the delayed model effectiveness verification, most business personnel only participate in the project for a period of time and then perform their regular duties. Their enthusiasm for participation diminishes as the project progresses, which is detrimental to the project schedule.

[0115] To address the aforementioned technical issues, the image recognition model construction method provided in this embodiment essentially offers a real-time self-learning-based intelligent pre-labeling platform. This platform, based on real-time automatic machine learning, overcomes the problems of inefficient labeling and slow training effect improvement in image recognition applications, providing a simplified user experience. By placing automatic machine learning technology at the forefront of the labeling process, it accelerates the labeling speed. Newly learned models replace older models in real-time for machine labeling (model labeling), continuously optimizing the labeling experience through machine learning. For example, for fixed-format tickets, a pre-trained model can be generated using only one labeled data sample, while for non-fixed-format tickets, a pre-trained model can be generated using only ten labeled data samples.

[0116] More importantly, users can observe the learning effect of the model during the annotation process. Unlike the traditional OCR modeling process (3 days of annotation - 1 day of training - 1 day of verification), which requires a week to verify the learning effect of the model, users can understand the optimization process of the model in real time and complete the verification of the model through annotation, which greatly improves the timeliness.

[0117] Specifically, based on the application of Automated Machine Learning (AutoML) technology in the OCR field, and coupled with a sophisticated end-to-end UI design, this intelligent pre-labeling platform tool based on real-time self-learning has been developed. This reduces the time required to build a typical OCR scenario from three months to one week. Standard certificate scenarios can be built in just one day. From a technical architecture perspective, the intelligent pre-labeling platform tool can be mainly divided into four parts: an intelligent pre-labeling mode switching module, a real-time model self-learning module, a service management module, and a data management module. Figure 2 As shown.

[0118] The intelligent pre-labeling mode switching module mainly includes the core logic for intelligently switching between manual labeling mode and model labeling mode, dynamically providing the most user-friendly labeling experience. For details, please refer to the previous section on switching between manual and model labeling modes. For example, in any training cycle, firstly, the recognition accuracy of the target labeled samples is obtained; then, based on the recognition accuracy of the target labeled samples, the model labeling effect of the first training model is determined; then, based on the model labeling effect, the target labeling mode is determined; when the target labeling mode is manual labeling mode, the user's labeled data for unlabeled samples is obtained, resulting in labeled samples. Subsequently, when the current labeling mode is manual labeling mode, it can also determine whether preset switching conditions are met; when the switching conditions are met, the mode is switched from manual labeling mode to model labeling mode, etc.

[0119] The real-time self-learning module is responsible for real-time model training, real-time performance statistics, and displaying the results to the annotation team to monitor whether the latest model learning results meet the standards. Specifically, this includes: localization model training, feature extraction, information extraction model training, and model evaluation. For details, please refer to the previous sections on building the initial model and updating the training model over multiple training cycles.

[0120] Service Management Module: Responsible for deploying and deploying pre-trained models. This may include localization, recognition, and information extraction services. For details, please refer to the previous section on updating and deploying the trained model obtained in each training cycle.

[0121] Data Management Module: Responsible for the unified management of sample image data, annotation data, model data, and log data.

[0122] In addition to the four main components, the architecture of the intelligent pre-labeling platform tool also includes: storage layer, service layer, presentation layer, and system integration.

[0123] Storage layer: The storage layer provides three types of storage services to the upper layer: object storage, relational database storage, and monitoring data storage.

[0124] Service layer: It is divided into various modules and uses basic services (authentication, Kubernetes namespace management, configuration center, monitoring center, etc.) provided by the Artificial Intelligence (AI) operating system Sage AIOS. Modules are accessed through RESTful interfaces.

[0125] Presentation layer: The front-end and back-end are developed separately and deployed as different services, with traffic distributed by the network management layer.

[0126] System integration: When the delivery personnel deploy the system to the customer's system, they determine how to integrate with the customer's system based on the customer's system infrastructure.

[0127] like Figure 3As shown, based on the aforementioned intelligent pre-annotation platform tool, users can open the annotation page in the tool's front end. The tool's backend queries the database for corresponding template data to obtain annotation templates. The intelligent pre-annotation platform tool then activates the manual annotation mode based on these templates, prompting the user to manually annotate a specified number of samples. The user begins annotating sample images in this manual annotation mode. For non-fixed format sample images, after the user manually annotates ten sample images, these manually annotated image samples are used for model training to build an initial OCR model, which is then deployed online, entering model annotation mode and initiating a multi-cycle model update training process. In this model annotation mode, the user is provided with model annotation results for subsequent unannotated image samples. The user can manually review these model annotation results, modify or confirm the annotated content, and then use the manually reviewed sample annotation results to continue updating the OCR recognition model. For example, the electronic device can receive the user's modification instructions for the model annotation results, record the annotation recognition results before modification and the modified annotation content, and then update the trained OCR recognition model based on this annotation modification information.

[0128] like Figure 4 As shown, in model annotation mode, the model annotation results and the user's modification results are returned to the backend, which stores them for calculating the accuracy of the model annotation results. If the average accuracy is less than 50% after annotating three images, the system switches to manual annotation mode, and the user continues to annotate manually. When a new OCR recognition model is obtained by updating the training based on the newly annotated data, the OCR recognition model can be deployed online. Then, the system switches back to model annotation mode, and the new model is used to automatically recognize the annotation data of the samples to be annotated.

[0129] During the annotation process, manual annotation mode and model annotation mode can be switched multiple times based on certain conditions. Specifically, switching from manual annotation mode to model annotation mode occurs when a new OCR recognition model has been developed; while switching from model annotation mode to manual annotation mode occurs when the machine annotation accuracy of the new OCR recognition model falls below a certain threshold. The annotation mode switching process ends after all sample images have been annotated.

[0130] The technical solution proposed in this embodiment supports the extraction of non-fixed format fields, making its application more widespread and faster. It compresses the modeling and usage process into intelligent annotation, updating the results in real time during annotation. Users can observe the latest results in real-time, and stop annotation once satisfied, thus completing the construction and use of the OCR model. Model annotation is also easier to operate than manual annotation. The system displays the detected annotation content on the image based on the latest self-learning model, and users only need to view and modify it to complete the annotation. Furthermore, the model annotation mode supports automatic switching between user behavior and manual annotation mode. Compared with traditional methods, applying the method of this embodiment can make model construction faster, reaching a level where model training is completed as soon as annotation ends. The amount of training data depends on the degree of image standardization; a fixed format requires only one image to train the model, while a non-fixed format requires ten images, supporting continuous optimization and iteration of the model.

[0131] Furthermore, as Figure 1 The specific implementation of the method shown in this embodiment provides an image recognition model construction device, such as... Figure 5 As shown, the device includes: a construction module 21 and a training module 22.

[0132] Module 21 is configured to build the initial model;

[0133] Training module 22 is configured to iteratively train the initial model; wherein the training process of any training cycle includes: iteratively training the first training model of the current training cycle using the labeled samples of the previous training cycle until a second training model is obtained; before the first training model completes its iterative training, using the first training model to label and identify the currently unlabeled samples to obtain the labeling and identification results; obtaining the manual review results of the labeling and identification results, and determining the labeled samples of the current training cycle based on the manual review results.

[0134] In a specific application scenario, the training module 22 is specifically configured to: obtain the recognition accuracy of the target labeled samples in any training cycle; determine the model labeling effect of the first training model based on the recognition accuracy of the target labeled samples; determine the target labeling mode based on the model labeling effect, wherein the target labeling mode includes: manual labeling mode and model labeling mode; when the target labeling mode is manual labeling mode, obtain the user's labeling data for unlabeled samples to obtain labeled samples.

[0135] In specific application scenarios, the training module 22 is further configured to obtain the recognition accuracy of the labeled samples in the current training period; or, obtain the recognition accuracy of all labeled samples; or, obtain the recognition accuracy of all labeled samples excluding those manually labeled.

[0136] In specific application scenarios, the training module 22 is further configured to determine the target annotation mode as the manual annotation mode when the model annotation effect does not meet the preset conditions; or, when the model annotation effect does not meet the preset conditions, output a prompt message to prompt the user whether to switch the annotation mode; and determine the target annotation mode based on the user's operation information in response to the prompt message.

[0137] In a specific application scenario, the training module 22 is further configured to determine whether a preset switching condition is met in any training cycle when the current annotation mode is manual annotation mode; when the switching condition is met, the manual annotation mode is switched to the model annotation mode.

[0138] In specific application scenarios, the training module 22 is further configured to determine that the switching conditions are met when the number of manually labeled data reaches a preset number, or when a user's mode switching instruction is received, or when a new training model is obtained through training the results of manually labeled data.

[0139] In a specific application scenario, the construction module 21 is specifically configured to obtain a labeling template, which is used to describe the labeling content to be labeled and recognized by the image recognition model; based on the labeling template, prompting the user to manually label a specified number of first image samples, and obtaining a second image sample containing manually labeled data; wherein, the specified number meets the minimum sample requirement for training the initial model; and using the second image sample to train the model to obtain the initial model.

[0140] In specific application scenarios, optionally, the annotation template includes: a fixed template and a non-fixed template; wherein, the fixed template is used to describe the image recognition model annotating and recognizing the annotation content according to a fixed annotation position; the non-fixed template is used to describe the image recognition model annotating and recognizing the annotation content at a non-fixed position in the image.

[0141] In a specific application scenario, the training module 22 is further configured to, in any training cycle, bring the second training model trained in the current training cycle online for use in labeling and recognizing unlabeled samples in the next training cycle, and obtain labeling and recognition results.

[0142] In specific application scenarios, the sample type may optionally include at least one of the following:

[0143] Images of identification documents, receipts, and invoices.

[0144] It should be noted that other corresponding descriptions of the functional units involved in the image recognition model construction apparatus provided in this embodiment can be found in [reference needed]. Figure 1 The corresponding description in [the document] will not be repeated here.

[0145] Figure 6 A schematic diagram of the structure of an electronic device provided in this embodiment is shown.

[0146] In some embodiments of this disclosure, Figure 6 The electronic devices shown can be mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), in-vehicle terminals (such as in-vehicle navigation terminals), wearable devices, etc., as well as fixed terminals such as digital TVs, desktop computers, smart home devices, etc.

[0147] like Figure 6 As shown, the electronic device may include a processor 31 and a memory 32 storing computer program instructions.

[0148] Specifically, the processor 31 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0149] Memory 32 may include a large-capacity storage device for information or instructions. For example, and not limitingly, memory 32 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 32 may include removable or non-removable (or fixed) media. Where appropriate, memory 32 may be internal or external to the integrated gateway device. In a particular embodiment, memory 32 is a non-volatile solid-state memory. In a particular embodiment, memory 32 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (Electrically Programmable ROM, EPROM), an electrically erasable programmable ROM (EEPROM), an electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0150] The processor 31 performs the steps of the tree structure indexing method provided in this embodiment by reading and executing computer program instructions stored in the memory 32.

[0151] In one example, the electronic device may also include a transceiver 33 and a bus 34. Wherein, as... Figure 6 As shown, the processor 31, memory 32 and transceiver 33 are connected via bus 34 and communicate with each other.

[0152] Bus 34 may be hardware, software, or both. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 34 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0153] This disclosure also provides a computer-readable storage medium that can store a computer program that, when executed by a processor, enables the processor to implement the image recognition model construction method provided in this disclosure.

[0154] The aforementioned storage medium may, for example, include a memory 32 containing computer program instructions, which can be executed by a processor 31 of a tree-structured indexing device to complete the image recognition model construction method provided in this embodiment. Optionally, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device.

[0155] Through the above description of the embodiments, those skilled in the art can clearly understand that this disclosure can be implemented using software plus necessary general-purpose hardware platforms, or it can be implemented through hardware. By applying the solution of this embodiment, non-fixed format field extraction is supported, making its application more widespread and faster. The modeling and usage process is compressed into intelligent annotation, where the annotation effect is updated in real time. Users can observe the latest effect in real time through the interactive interface, and stop annotation once satisfied with the effect, thus completing the construction and use of the image recognition model. Pre-annotation and training can be completed in as little as half a day. Furthermore, the intelligent pre-annotation mode supports automatic switching between user behavior and standard annotation modes, making it more intelligent and user-friendly.

[0156] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for constructing an image recognition model, comprising: The process of constructing an image recognition model includes: constructing an initial model and iteratively training the initial model; wherein, the training process in any training cycle includes: The labeled samples from the previous training cycle are used to iteratively train the first training model in the current training cycle until the second training model is obtained. Before the first training model completes its iterative training, the first training model is used to label and identify the currently unlabeled samples to obtain the labeling and identification results. Obtain the manual review results of the labeled recognition results, and determine the labeled samples for the current training period based on the manual review results; In any of the training cycles, the method further includes: Obtain the recognition accuracy of the labeled target samples, wherein the recognition accuracy of the labeled target samples is the degree of matching between the labeling recognition result of the first training model in the current training cycle and the manual review result; Based on the recognition accuracy of the labeled samples of the target, the model labeling effect of the first training model is determined; Based on the model annotation results, the target annotation mode is determined, wherein the target annotation mode includes: manual annotation mode and model annotation mode; When the target annotation mode is manual annotation mode, the annotation data of the user for the unannotated samples is obtained to obtain the labeled samples.

2. The method of claim 1, wherein, The acquisition of the recognition accuracy of the labeled target samples includes: Get the recognition accuracy of the labeled samples in the current training period; or, Obtain the recognition accuracy of all labeled samples; or, Obtain the recognition accuracy of all labeled samples excluding those manually labeled.

3. The method of claim 1, wherein, The process of determining the target annotation pattern based on the model annotation effect includes: When the model annotation effect does not meet the preset conditions, the target annotation mode is determined to be the manual annotation mode; or... When the model annotation effect does not meet the preset conditions, a prompt message is output to ask the user whether to switch the annotation mode; and the target annotation mode is determined based on the user's operation information in response to the prompt message.

4. The method of claim 1, wherein, In any of the training cycles, the method further includes: When the current annotation mode is manual annotation mode, determine whether the preset switching conditions are met; When the switching conditions are met, the manual annotation mode is switched to the model annotation mode.

5. The method of claim 4, wherein, The determination of whether the preset switching conditions are met includes: The switching condition is determined to be met when the number of manually labeled items reaches a preset number, or when a user's mode switching instruction is received, or when a new training model is obtained through training the results of manually labeled items.

6. The method according to any one of claims 1 to 5, characterized in that, The construction of the initial model includes: Obtain a labeling template, which describes the labeling content to be labeled and recognized by the image recognition model; Based on the annotation template, the user is prompted to manually annotate a specified number of first image samples and obtain second image samples containing manually annotated data; wherein, the specified number satisfies the minimum sample requirement for training the initial model; The initial model is obtained by training the model using the second image sample.

7. The method of claim 6, wherein, The annotation templates include: fixed-format templates and non-fixed-format templates; The fixed template is used to describe how the image recognition model identifies the labeled content according to fixed labeled positions. The non-fixed template is used to describe how the image recognition model annotates and recognizes the labeled content at non-fixed positions in the image.

8. The method according to any one of claims 1 to 5, characterized in that, In any of the training cycles, the method further includes: The second training model trained in the current training cycle is put online and used to label and identify unlabeled samples in the next training cycle, thus obtaining the labeling and identification results.

9. The method of claim 1, wherein, The types of samples include at least one of the following: Images of identification documents, receipts, and invoices. 10.A device for constructing an image recognition model, comprising: include: The building module is configured to build the initial model; The training module is configured to iteratively train the initial model; wherein the training process of any training cycle includes: iteratively training the first training model of the current training cycle using the labeled samples of the previous training cycle until a second training model is obtained; before the first training model completes its iterative training, using the first training model to label and identify the currently unlabeled samples, and obtaining the labeling and identification results; obtaining the manual review results of the labeling and identification results, and determining the labeled samples of the current training cycle based on the manual review results; The training module is further configured to, in any training cycle, obtain the recognition accuracy of the target labeled samples, wherein the recognition accuracy of the target labeled samples is the degree of matching between the labeling recognition result of the first training model and the manual review result in the current training cycle; based on the recognition accuracy of the target labeled samples, determine the model labeling effect of the first training model; based on the model labeling effect, determine the target labeling mode, wherein the target labeling mode includes: manual labeling mode and model labeling mode; when the target labeling mode is manual labeling mode, obtain the user's labeling data for unlabeled samples to obtain labeled samples.

11. An electronic device, comprising: include: processor; Memory, used to store executable instructions; The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the method of any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, causes the processor to implement the method described in any one of claims 1-9.

Images