An image recognition method, device and equipment based on a recognition model and a medium

Abstract

The application provides an image recognition method and device based on a recognition model, equipment and a medium, which are used in the technical field of image recognition and can solve the problem of low accuracy of existing feature recognition. The method comprises the following steps: detecting a to-be-recognized image according to a target detection model to obtain target images and labeled features, and determining a prompt template corresponding to each target image according to the feature category of the labeled features; filling the feature category into the prompt template to obtain a model prompt, and determining a model output according to the feature category and the corresponding labeled position of the labeled features; determining a model training data set according to the model prompt, the model output, and an interference prompt, and training an initial recognition model according to the model training data set to obtain a target recognition model; and recognizing an unlabeled image according to the target recognition model to obtain an image feature corresponding to the unlabeled image. Thus, the accuracy and efficiency of image recognition are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and in particular to an image recognition method and device based on a recognition model, an image recognition equipment and a medium. BACKGROUND

[0002] A file collection arranged and bound according to certain rules is called a dossier, which is a standardized archive carrier formed by various institutions in the process of carrying out business. Through image recognition, the positions and contents of key feature elements such as titles, case numbers, signatures and seals can be extracted from the dossier image, and unstructured images can be converted into structured data, thereby providing a data basis for subsequent retrieval and analysis.

[0003] The existing recognition method of a dossier image generally inputs the dossier image to determine the positions and categories of the elements in the dossier image. For example, for text elements, texture, shape features and deep learning methods can be used to extract corresponding image features.

[0004] However, the existing recognition method of a dossier image has the problems of low recognition efficiency and inaccurate recognition results. SUMMARY

[0005] The present application provides an image recognition method and device based on a recognition model to solve the problems of low recognition efficiency and inaccurate recognition results of the existing recognition method of a dossier image.

[0006] In a first aspect, the present application provides an image recognition method based on a recognition model, which comprises the following steps:

[0007] According to the target detection model, the to-be-recognized image is detected to obtain target images and at least one labeled feature corresponding to the target images, and according to the feature categories of the labeled features, the prompt template corresponding to each target image is determined;

[0008] The feature categories are filled into the corresponding prompt template to obtain the model prompt corresponding to each target image, and according to the feature categories and labeled positions of the labeled features in each target image, the model output is determined;

[0009] According to the model prompt, the model output and the interference prompt, the model training data set corresponding to each target image is determined, and the initial recognition model is trained according to the model training data set to obtain a target recognition model;

[0010] According to the target recognition model, the unlabeled image is recognized to obtain the image features corresponding to the unlabeled image.

[0011] In some embodiments of the present application, the target image and the at least one labeled feature corresponding to the target image are obtained by detecting the to-be-recognized image according to the target detection model, including:

[0012] The labeled image is input into the initial detection model to obtain a plurality of feature maps output by the initial detection model, and the preset number of feature maps are calculated according to the self-attention of the initial detection model to obtain a target feature map;

[0013] The target feature map and other feature maps that do not undergo self-attention calculation are fused to obtain a plurality of training feature maps, and the target detection model is determined according to the training feature maps and the initial detection model;

[0014] The target image and the labeled feature are obtained by detecting the to-be-recognized image according to the target detection model.

[0015] In some embodiments of the present application, the target detection model is determined according to the training feature maps and the initial detection model, including:

[0016] The detection frame corresponding to each training feature map is determined according to the initial detection model, and the distance loss function and the shape loss function between the labeled frame and the detection frame are calculated according to the labeled frame in the labeled image;

[0017] The corresponding loss function value is determined according to the distance loss function and the shape loss function, and the comparison result is obtained by comparing the numerical size of each loss function value and the preset loss threshold value;

[0018] If the comparison result is that each loss function value is greater than the preset loss threshold value, the initial detection model is determined as the target detection model;

[0019] If the comparison result is that there is at least one loss function value that is not greater than the preset loss threshold value, the model parameters of the initial detection model are adjusted according to the loss function value that is not greater than the preset loss threshold value to obtain the target detection model.

[0020] In some embodiments of the present application, the feature category is filled into the corresponding prompt template to obtain the model prompt corresponding to each target image, including:

[0021] A preset prompt template is determined, and a plurality of prompt templates are generated by performing similar language generation on the preset prompt template according to a language model;

[0022] The feature category is filled into the prompt template to obtain the model prompt.

[0023] In some embodiments of the present application, the model output language is determined according to the feature category and the corresponding labeled position of the labeled feature in each target image, including:

[0024] According to the bounding box corresponding to the labeled feature, a two-dimensional coordinate of the bounding box in the target image is determined, and according to the two-dimensional coordinate, a labeled position corresponding to the labeled feature is determined.

[0025] In some embodiments of the present application, according to the model prompt, the model output, and the interference prompt, a model training data set corresponding to each target image is determined, including:

[0026] From the target image, a plurality of target images and their corresponding feature categories are determined, and from a plurality of preset feature categories, an interference feature category different from the feature category is determined;

[0027] The interference feature category is filled into the prompt template to obtain an interference prompt, and according to the model prompt and the interference prompt, a prompt in the model training data set is determined.

[0028] In some embodiments of the present application, according to the model training data set, an initial recognition model is trained to obtain a target recognition model, including:

[0029] The optimization strategy of the initial recognition model is set to a ZeRO-3 optimization strategy, and according to the model training data set, the target image and the corresponding prompt are input to the initial recognition model to obtain an initial output of the initial recognition model;

[0030] According to the initial output and the model output corresponding to the target image, the model parameters of the initial recognition model are adjusted to obtain a target recognition model.

[0031] In a second aspect, the present application provides an image recognition device based on a recognition model, the device comprising:

[0032] A detection module is configured to detect a target image and at least one labeled feature corresponding to the target image according to a target detection model, and determine a prompt template corresponding to each target image according to the feature category of the labeled feature;

[0033] A determination module is configured to fill the feature category into the corresponding prompt template to obtain a model prompt corresponding to each target image, and determine a model output according to the feature category of the labeled feature and the corresponding labeled position in each target image;

[0034] A training module is configured to determine a model training data set corresponding to each target image according to the model prompt, the model output, and the interference prompt, and train an initial recognition model according to the model training data set to obtain a target recognition model;

[0035] A recognition module is configured to recognize an unlabeled image according to the target recognition model to obtain an image feature corresponding to the unlabeled image.

[0036] In a third aspect, the present application provides an apparatus, comprising: a processor, and a memory connected with the processor in communication;

[0037] The memory stores computer-executable instructions;

[0038] The processor executes the computer-executable instructions stored in the memory to implement the method of the present application.

[0039] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores program codes, and the program codes are executed by a processor to implement the method of the present application.

[0040] The present application provides an image recognition method, device and equipment based on a recognition model and a computer readable storage medium. The method comprises the following steps: detecting a target image and at least one labeled feature corresponding to the target image by using a target detection model; determining a prompt template corresponding to each target image according to a feature category of the labeled feature; filling the feature category into the corresponding prompt template to obtain a model prompt corresponding to each target image; determining a model output according to the feature category and a labeled position of the labeled feature in each target image; determining a model training data set corresponding to each target image according to the model prompt, the model output and an interference prompt; training an initial recognition model according to the model training data set to obtain a target recognition model; and recognizing an unlabeled image according to the target recognition model to obtain an image feature corresponding to the unlabeled image.

[0041] In this way, the labeled feature can be generated and matched with the prompt template by using the target detection model, and the model prompt and the output can be determined according to the category and position of the labeled feature, and the training data set can be constructed by introducing the interference prompt, so that the target recognition model obtained by training can not only significantly improve the positioning accuracy of the file element and solve the problem of inaccurate recognition, but also realize the recognition ability of various application scenarios through the interference prompt and the structured data, thereby avoiding overfitting caused by single data. BRIEF DESCRIPTION OF DRAWINGS

[0042] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0043] Figure 1 A flowchart of an image recognition method based on a recognition model provided by an embodiment of the present application is shown in the figure.

[0044] Figure 2 A flowchart of another image recognition method based on a recognition model provided by an embodiment of the present application is shown in the figure.

[0045] Figure 3 A structural schematic diagram of an image recognition device based on an identification model provided by an embodiment of the present application is shown in FIG. 1.

[0046] Figure 4 A structural block diagram of an apparatus for performing an image recognition method based on an identification model according to an embodiment of the present application is shown in FIG. 2. DETAILED DESCRIPTION

[0047] The exemplary embodiments will be described in detail herein with reference to the drawings. When the description below refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0048] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments. The embodiments of the present application will be described below with reference to the drawings.

[0049] Figure 1 A flowchart of an image recognition method based on an identification model provided by an embodiment of the present application is shown in FIG. 3. As shown in FIG. 3, the image recognition method based on an identification model can include the following steps: Figure 1

[0050] S110, detecting the to-be-recognized image according to a target detection model to obtain target images and at least one labeled feature corresponding to the target images, and determining a prompt template corresponding to each target image according to a feature category of the labeled feature.

[0051] The target detection model refers to a model for locating and identifying specific objects from an image, for example, a YOLO v10 model. By training and learning the visual features of various elements in the image file, the key objects (such as titles, seals, fingerprints, etc.) in the image are detected and the position and corresponding category of the object are outputted, so as to realize feature labeling and extraction.

[0052] The to-be-recognized image refers to an original image of a file that has not been processed or labeled, for example, an unlabeled scanned file of a file or an image of an approval form. These images can be used as input for model detection and contain various elements (such as text, seals, signatures, etc.) that need to be extracted, so that the detection model can perform image feature recognition to determine the position and category of the elements and perform labeling.

[0053] ​The target image refers to an image after feature recognition and labeling by the target detection model.

[0054] The labeled feature refers to key information extracted and labeled by the target detection model from the target image to describe the attribute of the element, including the element position (such as the bounding box coordinate bbox_2d) and the element existence (such as the is_Have field); for example, the model detects that there is a "seal" feature element in a target image, and labels the position of the feature to obtain the labeled feature "seal".

[0055] The feature category refers to a classification label of the element type to which the labeled feature belongs, used to distinguish different types of dossier elements; for example, different feature categories such as title, seal, fingerprint, and signature, wherein the title and seal belong to the category containing text content, and the fingerprint and picture belong to the category without text content; so as to determine different prompt templates and model output formats according to the different categories of features.

[0056] The prompt template refers to a pre-constructed text template used to guide the model to generate standardized output, containing a reserved element embedding position and a fixed format; for example, a template for the seal category: please locate the {feature category} in the image and output its position and text content, the template can be dynamically adapted according to the different feature categories to ensure the semantic consistency and format specification of the model training data.

[0057] Based on this, by determining the target detection model, the dossier image that needs to be recognized and labeled is recognized and labeled according to the target detection model to obtain the labeled target image, and the labeled features and feature categories corresponding to each target image, so as to determine the prompt templates corresponding to different categories according to the feature categories, and then use the prompt templates to guide the model to generate corresponding model outputs.

[0058] S120, fill the feature category into the corresponding prompt template to obtain the model prompt corresponding to each target image, and determine the model output according to the feature category and the corresponding labeled position of the labeled feature in each target image.

[0059] Among them, model prompts refer to natural language instructions generated after filling feature categories into prompt templates, which are used to guide the model in reasoning. They are used to help the model determine the corresponding recognition task through standardized text descriptions. In practical applications, prompts have a significant impact on the model's output results. By clarifying the task objectives, prompts guide the model to focus on key needs. The clarity of the content, the standardization of the format, and the diversity of scenarios directly affect the model's understanding accuracy of the feature localization task, the consistency of the output format, and its adaptability to complex scenarios. Therefore, by determining the model prompts, the model can adjust its output structure according to the format constraints of the prompts, thereby improving the model's recognition accuracy.

[0060] The location index indicates the specific spatial coordinates of the feature in the target image, which is usually represented by a two-dimensional bounding box (bbox_2d), that is, the pixel coordinates of the upper left corner (x1, y1) and lower right corner (x2, y2) of the rectangular area where the feature is located.

[0061] Model output refers to the standardized result that the model should output according to the task requirements. It is a response to the model prompts and contains key information related to the labeled features. The specific content of the output is determined by the type of feature category. Depending on whether the feature category contains text content, the model output is divided into two categories: For features without text content (such as fingerprints and images), the output includes bbox_2d (label location) and is_Have (whether the feature exists), for example: {bbox_2d:[100,200,200,300],is_Have:True}; For features with text content (such as titles and seals), the output additionally includes text (feature text content), for example: {bbox_2d:[50,50,400,100],is_Have:True,text:“Civil Case File”}.

[0062] Based on this, by determining a preset prompt template and filling the corresponding blanks in the prompt template with the feature categories of the labeled features in the target image, the corresponding model prompt is obtained. Furthermore, the model output is determined based on the feature categories and labeling positions of the labeled features. The model prompt and model output can be understood as the standard input and standard output of the model, so that they can be used for model training to obtain the target recognition model.

[0063] S130. Based on the model prompts, model outputs, and interference prompts, determine the model training dataset corresponding to each target image, and train the initial recognition model based on the model training dataset to obtain the target recognition model.

[0064] The interference prompt is a prompt word used to simulate a no-target element query scene, thereby enhancing the judgment ability of the recognition model on non-existing elements, complementing the model prompt (for real existing elements), and avoiding the situation where the model can only handle "element present" scenes and ignores "element absent" cases. For example, if a seal actually exists in the target image, the corresponding model prompt is to locate the seal in the image, and the interference prompt can be set as locating a fingerprint in the image (there is no fingerprint in the image), thereby enabling the model to learn how to respond to non-existing element queries, such as outputting no such element or empty coordinates, and improving the robustness of the model in complex scenes.

[0065] The model training data set is a structured data set formed by combining the target image, the model prompt, the model output word, and the interference prompt, which is used to train the recognition model, so that the model learns the mapping relationship between the image, the instruction, and the result through a large number of similar samples.

[0066] The initial recognition model is a basic recognition model, i.e., a pre-trained model with general image recognition capability.

[0067] The target recognition model is a final model obtained after fine-tuning of the model training data set, which can be used for the file element positioning task, thereby learning the training data of the file scene on the basis of the initial recognition model, and being used for element positioning and feature extraction of unlabeled file images.

[0068] Based on this, the model prompt corresponding to the target image is determined, and then the interference prompt for improving the robustness of the model is determined, so as to determine the model training data set according to the model prompt, the model output, and the interference prompt, and to train the initial recognition model according to the model training data set to obtain the target recognition model.

[0069] S140, according to the target recognition model, recognizing the unlabeled image to obtain image features corresponding to the unlabeled image.

[0070] The unlabeled image refers to an original file image that has not been labeled by humans or models, i.e., an image data without any element position, category, or content marked, which needs to be identified and labeled by the target recognition model.

[0071] The image features are key information automatically extracted by the target recognition model from the unlabeled image to describe the attributes of the elements in the image, such as the specific position and content of the element features in the image.

[0072] Based on this, the target recognition model is determined, and in actual application, the unlabeled image that needs to be identified and labeled is processed by the target recognition model to obtain the corresponding element features.

[0073] Based on the feasible implementation of S110 described above, this application further provides a method for detecting an image to be recognized according to a target detection model, obtaining a target image and at least one labeled feature corresponding to the target image, including:

[0074] The labeled image is input into the initial detection model to obtain multiple feature maps output by the initial detection model. Based on the initial detection model, self-attention calculation is performed on a preset number of feature maps to obtain the target feature map.

[0075] The target feature map and other feature maps that have not undergone self-attention calculation are fused to obtain multiple training feature maps. Based on the training feature maps and the initial detection model, the target detection model is determined.

[0076] Based on the target detection model, the image to be identified is detected to obtain the target image and labeled features.

[0077] The initial detection model refers to the basic model used for feature extraction and preliminary detection. It is usually a pre-trained convolutional neural network (such as the basic network part of ResNet and YOLO), which has the ability to extract multi-level visual features from images, but has not yet been optimized for specific tasks (such as dossier feature localization).

[0078] Annotated images refer to images that have been manually or automatically labeled with key information.

[0079] Feature maps refer to the two-dimensional tensors output by each layer of the initial detection model. They can be understood as abstract representations of visual features at different levels of the input image. For example, they can be shallow feature maps containing low-level features such as edges and textures (e.g., the texture of the paper in a file, the stroke edges of text); or they can be deep feature maps containing high-level features such as semantics and categories (e.g., the overall outline of a seal, the text region features of a title). Feature maps are the intermediate carriers for the model to understand the content of an image. Feature maps at different levels reflect different dimensions of information about the image, from details to the whole.

[0080] The preset quantity refers to the number of feature maps that are manually set and require self-attention calculation.

[0081] Self-attention computation is a mechanism that allows a model to focus on important regions in a feature map. By calculating the association weights of each location within the feature map, it enhances the representation of key information and suppresses irrelevant information.

[0082] The target feature map refers to the optimized feature map obtained after performing self-attention calculation on a preset number of feature maps. Compared with the original feature map, the target feature map strengthens the feature representation of key elements and reduces noise interference through weight adjustment, and can better reflect the essential attributes of core elements in the image (such as the color and shape features of the seal).

[0083] Training feature maps refer to the set of feature maps obtained by fusing the target feature map with other feature maps that have not undergone self-attention calculation. This fusion can be achieved through methods such as concatenation or weighted summation, thereby preserving the advantages of features at different levels.

[0084] Based on this, the data is first labeled, for example, using a visual labeling tool for preliminary data labeling. The labeled images are then input into the initial recognition model for training. Multiple feature maps output by the initial detection model are used to perform self-attention calculations on some feature maps, which are then fused with another part. This partial self-attention calculation reduces computational overhead. Furthermore, model training can also combine partial parameter training and full parameter training. Specifically, partial parameter training is used in the first two-thirds of the training batches, training only the classification layer weights while keeping the feature extraction layer weights unchanged. Full parameter training is used in the last one-third of the training batches, training all parameters of the model. The specific number of training rounds depends on the amount of data. This yields the trained target recognition model.

[0085] Based on the feasible implementation of S110 described above, this application further provides a method for determining a target detection model based on the trained feature map and the initial detection model, including:

[0086] Based on the initial detection model, the detection boxes corresponding to each training feature map are determined, and the distance loss function and shape loss function between the annotation boxes and the detection boxes are calculated based on the annotation boxes in the labeled images.

[0087] Based on the distance loss function and the shape loss function, the corresponding loss function values ​​are determined, and the values ​​of each loss function value are compared with the preset loss threshold to obtain the comparison results;

[0088] If the comparison results show that all loss function values ​​are greater than the preset loss threshold, then the initial detection model is determined to be the target detection model.

[0089] If the comparison result shows that there is at least one loss function value that is not greater than the preset loss threshold, then the model parameters of the initial detection model are adjusted according to the loss function value that is not greater than the preset loss threshold to obtain the target detection model.

[0090] The detection box refers to the bounding box of the predicted feature location after the initial detection model infers from the training feature map. It is usually represented by two-dimensional coordinates [x1, y1, x2, y2] (pixel coordinates of the upper left and lower right corners). For example, after the model detects a seal in a case file image, the output [150, 300, 350, 500] is the detection box, which represents the region where the predicted feature is located.

[0091] The annotation box refers to the actual location bounding box of manually or semi-automatically annotated elements in the image.

[0092] The distance loss function is a mathematical function used to quantify the spatial positional deviation between the detection box and the annotation box, measuring the distance difference between the two in the image.

[0093] The shape loss function is a mathematical function used to quantify the geometric shape difference between the detection box and the annotation box, measuring the deviation between the two in terms of aspect ratio and overall outline.

[0094] The preset loss threshold refers to a pre-set critical value of the loss function, which is used to determine whether the current training effect of the model meets the standard, thereby measuring the loss gap between the matching accuracy of the detection box and the annotation box.

[0095] Based on this, by determining the distance loss function and shape loss function between the detection box generated by the target detection model and the corresponding annotation box of the target image, the corresponding loss function value can be further determined. For example, the loss function value can be obtained by weighted calculation. By comparing the loss function value with the preset loss threshold, the model parameters of the detection model can be determined according to the comparison result. If they are not qualified, the model parameters need to be further adjusted to obtain the target detection model.

[0096] Based on the feasible implementation of S120 described above, this application further provides a method for filling feature categories into corresponding prompt templates to obtain model prompts for each target image, including:

[0097] A preset prompt template is determined, and similar words are generated from the preset prompt template based on the language model to obtain multiple prompt templates;

[0098] Fill the feature categories into the prompt template to obtain the model prompt.

[0099] Among them, the preset prompt template refers to the pre-designed basic text framework used to generate standardized prompts. It includes fixed formats and replaceable element placeholders, thereby providing a unified instruction structure for locating file elements and ensuring the standardization and task orientation of the prompts.

[0100] A language model refers to a natural language processing model that has the ability to generate text and understand semantics, thereby expanding more diverse prompts based on preset templates.

[0101] Similar word generation refers to the process by which a language model generates diverse prompts that have the same semantics as the template but are expressed differently, based on a preset prompt template. This enriches the diversity of prompts, avoids overfitting of the model during training due to the monotonous format of prompts, and improves the adaptability of the recognition model to various real-world scenarios, thus enhancing its scalability.

[0102] Based on this, in practical applications, a certain number of prompt word templates can be predetermined, and a large language model can be used for semantic transformation to obtain multiple prompt word templates.

[0103] Based on the feasible implementation of S120 described above, this application further provides a method for determining the model output based on the feature category and corresponding annotation position of the annotated features in each target image, including:

[0104] Based on the detection box corresponding to the labeled features, determine the two-dimensional coordinates of the detection box in the target image, and based on the two-dimensional coordinates, determine the label position corresponding to the labeled features.

[0105] Two-dimensional coordinates refer to the spatial position parameters of the detection box within the target image plane. They are used to accurately mark the specific areas of labeled features (such as seals, titles, etc.) in the image and are usually expressed in pixels.

[0106] Based on this, by determining the detection box of the target recognition model for the labeled features, the labeling position of the labeled features can be further determined according to the two-dimensional coordinates corresponding to the detection box.

[0107] Based on the feasible implementation of S130 described above, this application further provides a model training dataset for determining each target image based on model prompts, model outputs, and interference prompts, including:

[0108] From the target image, determine multiple target images and their corresponding feature categories, and from multiple preset feature categories, determine interference feature categories that are different from the feature categories;

[0109] The interference feature categories are filled into the prompt template to obtain the interference prompts. Based on the model prompts and the interference prompts, the prompts in the model training dataset are determined.

[0110] Among them, the preset feature categories refer to a predefined set of element types that cover all possible element types in the case file scene. It can be understood as a complete list of elements that the model needs to learn to recognize. It includes both the element categories that actually exist in the current target image and the element categories that may appear in the scene but are not included in the current image, in order to build a comprehensive element system.

[0111] Interference feature categories refer to feature types selected from preset feature categories that are different from the feature categories that actually exist in the current target image. In other words, feature categories that do not exist in the current image are used to simulate scenarios where users query for non-existent features in real-world applications, thereby enhancing the model's ability to judge cases where no features exist.

[0112] Based on this, in practical applications, users may only query features that exist in an image, or they may query features that do not exist. By introducing unlabeled features during the model training phase, the "querying features that do not exist" scenario that may occur in practical applications can be simulated, avoiding overfitting to a single pattern and improving the accuracy and robustness of the model recognition.

[0113] Based on the feasible implementation of S130 described above, this application further provides a method for training an initial recognition model using a model training dataset to obtain a target recognition model, including:

[0114] The initial recognition model is set to the ZeRO-3 optimization strategy. Based on the model training dataset, the target image and the corresponding prompt are input into the initial recognition model to obtain the initial output of the initial recognition model.

[0115] Based on the initial output and the model output corresponding to the target image, the model parameters of the initial recognition model are adjusted to obtain the target recognition model.

[0116] Among them, the ZeRO-3 optimization strategy refers to the ZeRO-3 (Zero Redundancy Optimizer Stage 3) optimization strategy, which is a memory optimization technique for large-scale model training. Through fine-grained parameter sharding and memory management, it solves the memory bottleneck problem during large model training and supports efficient model training under limited hardware resources.

[0117] Based on this, by determining the corresponding optimization strategy during model training, the initial output of the target recognition model for the target image and the prompt is obtained. This allows for further adjustment of the model parameters based on the initial output and the model output. For example, the model parameters can be adjusted using the loss function between the initial output and the model output to obtain the target recognition model, enabling it to output the model output based on the prompt corresponding to the target image.

[0118] Please refer to Figure 2 , Figure 2 A flowchart illustrating another image recognition method based on a recognition model provided in this application embodiment; as shown Figure 2 As shown, the images in the case file are manually annotated, and then the annotated data is used to train an object detection model. If the model training fails, it is retrained; if it succeeds, automatic annotation is performed based on the object detection model. Next, the automatically annotated data is manually corrected. If the corrected data passes the check, the final annotated data is obtained; if it fails, manual correction must be performed again.

[0119] In some embodiments of this application, an initial detection model is trained using labeled images to determine a target detection model. The target image to be identified is detected based on the target detection model, and the target image and its corresponding labeled features are obtained. The corresponding prompt template is determined based on the feature category of the labeled features, and the feature category is filled into the template to obtain the model prompt. The model output text can be determined by combining the feature category and label position of the labeled features. Based on the model prompt, model output text and interference prompt, a model training dataset corresponding to each target image is generated. The initial recognition model is then trained using the model training dataset to obtain a target recognition model. The target recognition model is then used to recognize unlabeled images and obtain their corresponding image features, thereby achieving accurate localization and structured feature extraction of case file image elements.

[0120] In this way, the target detection model can generate labeled features and match prompt templates. By combining the category and location of the labeled features, the model prompts and output messages can be determined. At the same time, interference prompts are introduced to build a training dataset. The target recognition model obtained after training can not only significantly improve the accuracy of case file element positioning and solve the problem of inaccurate recognition, but also achieve the ability to recognize various application scenarios through interference prompts and structured data, thus avoiding overfitting caused by single data.

[0121] Figure 3 This is a schematic diagram of the structure of an image recognition device 300 based on a recognition model, provided in an embodiment of this application. Figure 3 As shown, the image recognition device 300 based on a recognition model includes: a detection module 310, a determination module 320, a training module 330, and a recognition module 340; wherein:

[0122] The detection module 310 is used to detect the image to be identified according to the target detection model, obtain the target image and at least one labeled feature corresponding to the target image, and determine the prompt template corresponding to each target image according to the feature category of the labeled feature.

[0123] The determination module 320 is used to fill the feature categories into the corresponding prompt templates to obtain the model prompts for each target image, and to determine the model output text based on the feature categories and corresponding annotation positions of the labeled features in each target image.

[0124] The training module 330 is used to determine the model training dataset corresponding to each target image based on the model prompts, model outputs, and interference prompts, and to train the initial recognition model based on the model training dataset to obtain the target recognition model.

[0125] The recognition module 340 is used to recognize the unlabeled image according to the target recognition model and obtain the image features corresponding to the unlabeled image.

[0126] In this embodiment of the application, the detection module 310 can also be specifically used for:

[0127] The labeled image is input into the initial detection model to obtain multiple feature maps output by the initial detection model. Based on the initial detection model, self-attention calculation is performed on a preset number of feature maps to obtain the target feature map.

[0128] The target feature map and other feature maps that have not undergone self-attention calculation are fused to obtain multiple training feature maps. Based on the training feature maps and the initial detection model, the target detection model is determined.

[0129] Based on the target detection model, the image to be identified is detected to obtain the target image and labeled features.

[0130] In this embodiment of the application, the detection module 310 can also be specifically used for:

[0131] Based on the initial detection model, the detection boxes corresponding to each training feature map are determined, and the distance loss function and shape loss function between the annotation boxes and the detection boxes are calculated based on the annotation boxes in the labeled images.

[0132] Based on the distance loss function and the shape loss function, the corresponding loss function values ​​are determined, and the values ​​of each loss function value are compared with the preset loss threshold to obtain the comparison results;

[0133] If the comparison results show that all loss function values ​​are greater than the preset loss threshold, then the initial detection model is determined to be the target detection model.

[0134] If the comparison result shows that there is at least one loss function value that is not greater than the preset loss threshold, then the model parameters of the initial detection model are adjusted according to the loss function value that is not greater than the preset loss threshold to obtain the target detection model.

[0135] In this embodiment of the application, the determining module 320 may also be specifically used for:

[0136] A preset prompt template is determined, and similar words are generated from the preset prompt template based on the language model to obtain multiple prompt templates;

[0137] Fill the feature categories into the prompt template to obtain the model prompt.

[0138] In this embodiment of the application, the determining module 320 may also be specifically used for:

[0139] Based on the detection box corresponding to the labeled features, determine the two-dimensional coordinates of the detection box in the target image, and based on the two-dimensional coordinates, determine the label position corresponding to the labeled features.

[0140] In this embodiment of the application, the training module 330 can also be specifically used for:

[0141] From the target image, determine multiple target images and their corresponding feature categories, and from multiple preset feature categories, determine interference feature categories that are different from the feature categories;

[0142] The interference feature categories are filled into the prompt template to obtain the interference prompts. Based on the model prompts and the interference prompts, the prompts in the model training dataset are determined.

[0143] In this embodiment of the application, the training module 330 can also be specifically used for:

[0144] The initial recognition model is set to the ZeRO-3 optimization strategy. Based on the model training dataset, the target image and the corresponding prompt are input into the initial recognition model to obtain the initial output of the initial recognition model.

[0145] Based on the initial output and the model output corresponding to the target image, the model parameters of the initial recognition model are adjusted to obtain the target recognition model.

[0146] Figure 4 This is a schematic diagram of the structure of a device for performing an image recognition method based on a recognition model according to an embodiment of this application. Figure 4 As shown, the device 400 includes:

[0147] The device 400 may include a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a communication component 403, and other components. The processor 401, memory 402, and communication component 403 are connected via a bus 404.

[0148] In the specific implementation process, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to execute the above-mentioned image recognition method based on a recognition model.

[0149] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0150] Furthermore, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0151] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0152] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0153] In some embodiments, a computer program product is also provided, comprising a computer program or instructions that, when executed by a processor, implement the steps in any of the above-described image recognition methods based on recognition models.

[0154] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0155] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0156] Therefore, embodiments of this application provide a computer-readable storage medium storing multiple lines of program code that can be loaded by a processor to execute steps in any of the image recognition methods based on recognition models provided in embodiments of this application.

[0157] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0158] According to one aspect of this application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium.

[0159] Since the instructions stored in the storage medium can execute the steps of any of the image recognition methods based on recognition models provided in the embodiments of this application, the beneficial effects that any of the image recognition methods based on recognition models provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0160] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the appended claims.

[0161] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. An image recognition method based on a recognition model, characterized in that, The method includes: Based on the target detection model, the image to be identified is detected to obtain the target image and at least one labeled feature corresponding to the target image. Based on the feature category of the labeled feature, the prompt template corresponding to each target image is determined. The feature categories are filled into the corresponding prompt templates to obtain the model prompts for each target image. The model output message is determined based on the feature categories and corresponding annotation positions of the labeled features in each target image. Based on the model prompts, the model outputs, and the interference prompts, the model training dataset corresponding to each target image is determined, and the initial recognition model is trained based on the model training dataset to obtain the target recognition model; the interference prompts refer to prompts used to simulate a scenario of querying without target elements, thereby enhancing the recognition model's ability to judge non-existent elements and complementing the model prompts. Based on the target recognition model, the unlabeled image is identified to obtain the image features corresponding to the unlabeled image.

2. The method according to claim 1, characterized in that, The step of detecting the image to be identified according to the target detection model to obtain the target image and at least one labeled feature corresponding to the target image includes: A labeled image is input into an initial detection model to obtain multiple feature maps output by the initial detection model. Based on the initial detection model, a preset number of feature maps are subjected to self-attention calculation to obtain a target feature map. The target feature map and other feature maps that have not undergone self-attention calculation are fused to obtain multiple training feature maps, and the target detection model is determined based on the training feature maps and the initial detection model. The target image to be identified is detected according to the target detection model to obtain the target image and the labeled features.

3. The method according to claim 2, characterized in that, The step of determining the target detection model based on the trained feature map and the initial detection model includes: Based on the initial detection model, the detection boxes corresponding to each of the training feature maps are determined, and based on the annotation boxes in the labeled image, the distance loss function and shape loss function between the annotation boxes and the detection boxes are calculated; Based on the distance loss function and the shape loss function, the corresponding loss function values ​​are determined, and the values ​​of each loss function value are compared with the preset loss threshold to obtain the comparison results; If the comparison result shows that all the loss function values ​​are greater than the preset loss threshold, then the initial detection model is determined to be the target detection model; If the comparison result shows that at least one of the loss function values ​​is not greater than the preset loss threshold, then the model parameters of the initial detection model are adjusted according to the loss function values ​​that are not greater than the preset loss threshold to obtain the target detection model.

4. The method according to claim 1, characterized in that, The step of filling the feature categories into the corresponding prompt templates to obtain the model prompts for each target image includes: A preset prompt template is determined, and similar words are generated from the preset prompt template based on the language model to obtain multiple prompt templates; The feature category is filled into the prompt template to obtain the model prompt.

5. The method according to claim 1, characterized in that, The step of determining the model output based on the feature category and corresponding annotation position of the labeled features in each of the target images includes: Based on the detection box corresponding to the labeled feature, determine the two-dimensional coordinates of the detection box in the target image, and based on the two-dimensional coordinates, determine the label position corresponding to the labeled feature.

6. The method according to claim 1, characterized in that, The step of determining the model training dataset corresponding to each target image based on the model prompts, the model outputs, and interference prompts includes: From the target image, determine multiple target images and all their corresponding feature categories, and from multiple preset feature categories, determine interference feature categories that are different from the feature categories; The interference feature categories are filled into the prompt template to obtain the interference prompt, and the prompts in the model training dataset are determined based on the model prompt and the interference prompt.

7. The method according to claim 1, characterized in that, The step of training the initial recognition model based on the model training dataset to obtain the target recognition model includes: The optimization strategy of the initial recognition model is set to the ZeRO-3 optimization strategy. Based on the model training dataset, the target image and the corresponding prompt are input into the initial recognition model to obtain the initial output message of the initial recognition model. Based on the initial output and the model output corresponding to the target image, the model parameters of the initial recognition model are adjusted to obtain the target recognition model.

8. An image recognition device based on a recognition model, characterized in that, The device includes: The detection module is used to detect the image to be identified according to the target detection model, obtain the target image and at least one labeled feature corresponding to the target image, and determine the prompt template corresponding to each target image according to the feature category of the labeled feature; The determination module is used to fill the feature category into the corresponding prompt template to obtain the model prompt corresponding to each target image, and to determine the model output message based on the feature category and corresponding annotation position of the labeled features in each target image; The training module is used to determine the model training dataset corresponding to each target image based on the model prompts, the model outputs, and interference prompts, and to train the initial recognition model based on the model training dataset to obtain the target recognition model; the interference prompts refer to prompts used to simulate a query scenario without target elements, thereby enhancing the recognition model's ability to judge non-existent elements and complementing the model prompts. The recognition module is used to recognize the unlabeled image according to the target recognition model and obtain the image features corresponding to the unlabeled image.

9. A device, characterized in that, include: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in memory and configured to be executed by one or more processors, the one or more programs being configured to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that can be called by a processor to perform the method as described in any one of claims 1 to 7.

Images