Image labeling method and device, storage medium and electronic equipment

Abstract

The invention discloses an image annotation method and device, a storage medium and electronic equipment, and relates to the field of artificial intelligence. The method comprises the following steps: acquiring S target object images; determining target semantic feature information of the S target object images based on the target cue word; inputting the target semantic feature information of each target object image into a target model, and determining a label of each target object image according to the target model; and determining a target annotation image set according to the label of each target object image. The technical problem that in the prior art, automatic labeling precision on a fine-grained small sample data set is insufficient is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and more specifically, to an image annotation method, device, storage medium, and electronic device. Background Art

[0002] With the increasing sophistication of financial internal control and compliance management, the need for fine-grained classification of behaviors and individuals in surveillance videos is becoming increasingly prominent. However, traditional automated labeling technologies often suffer from insufficient recognition accuracy when faced with limited, small-sample datasets. Existing rule-based or pre-trained model-based approaches are particularly struggling to address subtle differences in banking environments—such as distinguishing the attire of bank employees, customers, and security personnel—resulting in high misclassification rates. Furthermore, while weakly supervised learning can leverage unlabeled data, it still faces challenges with poor generalization and overfitting in small-sample environments, making it difficult to maintain stable classification performance in complex scenarios.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] The present application provides an image annotation method, device, storage medium and electronic device to at least solve the technical problem of insufficient accuracy of automatic annotation on fine-grained small sample data sets in the prior art.

[0005] According to one aspect of the present application, an image annotation method is provided, comprising: acquiring S target object images, where S is an integer greater than or equal to 1; determining target semantic feature information of the S target object images based on a target prompt word, wherein the target prompt word includes a question text for distinguishing objects of different categories, and the target semantic feature information is used to characterize the answer information of each target object image based on the question text in the target prompt word; inputting the target semantic feature information of each target object image into a target model, and determining a label for each target object image based on the target model, wherein the label of the target object image is used to describe the type information of the target object in the target object image, and the target model is a classification model obtained by training an initial model based on the semantic feature information of images of different categories; and determining a target annotated image set based on the label of each target object image.

[0006] Optionally, the target model is trained by the following steps: obtaining an object image set and a target list, wherein the object image set includes M object images, and the question instructions included in the target list are used to distinguish the characteristics of objects of different categories, wherein M is an integer greater than 1; constructing a target prompt word based on the question instruction in the target list; iteratively training the initial model according to the object image set and the target prompt word to obtain the target model.

[0007] Optionally, obtaining a set of object images includes: obtaining N images, where N is an integer greater than or equal to 1; detecting objects in each image through a first model, and intercepting an image area where the object is located in each image, wherein the first model is used to identify objects in the image; and forming a set of object images according to the image areas corresponding to the objects in each image.

[0008] Optionally, the initial model is iteratively trained according to the object image set and the target prompt word to obtain the target model, including: inputting the object image set and the target prompt word into the multimodal large model, and generating semantic feature information of each object image in the object image set based on the multimodal large model; converting the semantic feature information of each object image in the object image set into a semantic feature list, and forming a semantic feature list set according to the semantic feature list of each object image, wherein the information in each semantic feature list corresponds one-to-one to the question instruction in the target list; the initial model is iteratively trained according to the object image set, the semantic feature list set and the metadata features to obtain the target model, wherein the metadata features are used to represent the temporal information and spatial information related to the image.

[0009] Optionally, the initial model is iteratively trained according to the object image set, the semantic feature list set and the metadata features to obtain a target model, including: determining target model parameters of the initial model, and updating the initial model to a second model based on the target model parameters, wherein the target model parameters are model hyperparameters determined based on a cross-validation method; obtaining a second set, wherein the second set includes the actual category of each object image in the object image set; constructing a training data set according to the second set, the semantic feature list set and the metadata features, wherein the second set is used as a training label set, and the semantic feature list set and the metadata features are used as a training sample set; iteratively training the second model according to the training data set until the number of iterations of the second model is greater than a preset number or the training error of the second model is lower than a set threshold, thereby obtaining a target model.

[0010] Optionally, each iterative training includes the following steps: predicting the training sample set through the second model to obtain a first prediction result corresponding to each training sample in the training sample set; determining the error value between the first prediction result corresponding to each training sample and the training label corresponding to each training sample; constructing a target decision tree based on the error value between the first prediction result corresponding to each training sample and the training label corresponding to each training sample, wherein the target decision tree is used to correct the error value; and updating the second model based on the target decision tree.

[0011] Optionally, updating the second model based on the target decision tree includes: training the target decision tree based on the error value between the first prediction result corresponding to each training sample and the training label corresponding to each training sample; predicting the training samples through the trained target decision tree to obtain the second prediction result corresponding to each training sample; merging the first prediction result and the second prediction result corresponding to each training sample to obtain the target prediction result corresponding to each training sample; and updating the second model based on the target prediction result.

[0012] According to another aspect of the present application, an image annotation device is also provided, including: a first acquisition unit, used to acquire S target object images, wherein S is an integer greater than or equal to 1; a first determination unit, used to determine target semantic feature information of the S target object images based on a target prompt word, wherein the target prompt word includes a question text for distinguishing objects of different categories, and the target semantic feature information is used to characterize the answer information of each target object image based on the question text in the target prompt word; a second determination unit, used to input the target semantic feature information of each target object image into a target model, and determine a label of each target object image based on the target model, wherein the label of the target object image is used to describe the type information of the target object in the target object image, and the target model is a classification model obtained by training an initial model based on the semantic feature information of images of different categories; a third determination unit, used to determine a target annotated image set based on the label of each target object image.

[0013] According to another aspect of the present application, a computer-readable storage medium is provided. The computer-readable storage medium includes a stored executable program, wherein when the executable program runs, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned image annotation method.

[0014] According to another aspect of the present application, an electronic device is also provided, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned image annotation method.

[0015] According to another aspect of an embodiment of the present application, a computer program product is further provided, including computer instructions, which implement the steps of the above-mentioned image annotation method when executed by a processor.

[0016] In the present application, S target object images are first acquired, where S is an integer greater than or equal to 1. Target semantic feature information of the S target object images is then determined based on a target prompt word, wherein the target prompt word includes a question text for distinguishing objects of different categories, and the target semantic feature information is used to characterize the answer information of each target object image based on the question text in the target prompt word. The target semantic feature information of each target object image is then input into a target model, and a label of each target object image is determined based on the target model, wherein the label of the target object image is used to describe the type information of the target object in the target object image, and the target model is a classification model obtained by training an initial model based on the semantic feature information of images of different categories. Finally, a target annotated image set is determined based on the label of each target object image. That is, by integrating the fine-grained feature extraction guided by the target prompt word with the classification model training method, the purpose of achieving accurate image annotation in a fine-grained small sample dataset is achieved, thereby achieving the technical effect of improving the accuracy of automatic annotation, especially effectively annotating fine-grained categories on a small sample dataset, thereby solving the technical problem of insufficient accuracy of automatic annotation on a fine-grained small sample dataset in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 is a flowchart of an optional image annotation method according to an embodiment of the present application;

[0019] Figure 2 is a schematic diagram of an optional image annotation method according to an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of an optional image annotation device according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] It should be noted that the collected information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation portals for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions to provide users with corresponding operation portals for users to choose to agree or refuse the automated decision-making results; if the user chooses to refuse, the expert decision-making process will be entered.

[0024] According to an embodiment of the present application, a method embodiment of an image annotation method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0025] It should be noted that an intelligent annotation system can be used as the execution subject of the image annotation method of the embodiment of the present application. It is understandable that the image annotation method provided in the embodiment of the present application can also be executed by other systems or devices, and the embodiment of the present application does not specifically limit this.

[0026] Figure 1 is a flow chart of an optional image annotation method according to an embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0027] Step S101: Acquire S target object images.

[0028] In step S101 , S is an integer greater than or equal to 1.

[0029] Optionally, the target object image refers to an image including the object that is cut out from the complete image.

[0030] Step S102: determining target semantic feature information of S target object images based on the target prompt word.

[0031] In step S102 , the target prompt word includes question text for distinguishing objects of different categories, and the target semantic feature information is used to represent the answer information of each target object image based on the question text in the target prompt word.

[0032] Optionally, the target prompt is a text consisting of a series of questions that guide the subsequent multimodal large model to extract the semantic features of the target object. For example, to distinguish between "customer" and "security guard", the target prompt may include questions such as "Is the target carrying a bag" or "Is the target wearing a uniform?"

[0033] Optionally, the intelligent tagging system obtains an image description through multimodal large model reasoning, which reflects the answer information of the question in the target prompt word, which can be specific statements such as "the target is wearing a white shirt" or "the target is carrying a backpack", for subsequent fine-grained classification.

[0034] Optionally, after acquiring the target object image, the intelligent annotation system feeds the target object image into a large multimodal model along with the target prompt word for processing. The large multimodal model can understand the question in the prompt word and generate a corresponding answer based on the image content, forming the target semantic feature information.

[0035] Optionally, through this process, the system can extract meaningful information from the image, which goes beyond simple visual features and includes descriptions of the identity, behavior or state of the target object, providing a richer and more concrete basis for subsequent classification.

[0036] Step S103: inputting the target semantic feature information of each target object image into the target model, and determining the label of each target object image according to the target model.

[0037] In step S103 , the label of the target object image is used to describe the type information of the target object in the target object image, and the target model is a classification model obtained by training the initial model based on the semantic feature information of images of different categories.

[0038] Optionally, the target model refers to a GBDT (Gradient Boosting Decision Tree) classifier, which is trained using a small sample of manually labeled data and is specifically designed to convert semantic feature information into fine-grained category labels.

[0039] Alternatively, GBDT is a decision tree-based ensemble learning method that achieves efficient classification and regression by gradually optimizing the prediction error. It has the advantage of handling complex feature spaces and is well-suited for structured data, especially tabular data. For datasets with a large number of features, GBDT algorithms can assess feature importance and select important features for classification, while less important features are used or not used at all.

[0040] Optionally, the tag is used to describe the type of the target object in the target object image, such as "bank clerk", "customer" or "security guard".

[0041] Optionally, the intelligent labeling system passes the target semantic feature information, along with relevant metadata (such as the image capture time, the position and size of the detection box, etc.), as input to the target model (GBDT classifier). The target model uses the patterns and regularities it has learned from a small amount of manually annotated data to assign a fine-grained category label to each target object image based on the input feature information.

[0042] Optionally, this process achieves the mapping from semantic feature information to specific type labels, which is the key to fine-grained classification. Through the decision tree structure of the GBDT model, feature information can be effectively filtered and combined to improve the accuracy and robustness of classification.

[0043] Step S104 : determining a target annotated image set according to the label of each target object image.

[0044] Optionally, a target annotated image set: a set consisting of all target object images that have been annotated with fine-grained category labels, is the output of the intelligent annotation system and can be directly used for subsequent data analysis, model training, or internal control and compliance management tasks.

[0045] Optionally, the intelligent labeling system will integrate the target object images and their corresponding fine-grained category labels to construct a target labeled image set, in which each image is clearly labeled with the fine-grained category information of the target object.

[0046] Optionally, this embodiment is applicable to various scenarios requiring fine-grained target detection. For example, in the financial industry, this embodiment can assist with internal control and compliance management, such as ensuring information security through illegal screen capture detection, improving security through security guard absence detection, and ensuring transaction transparency through surrogate operation detection, thereby comprehensively improving the automation level and regulatory efficiency of the financial system.

[0047] As can be seen from the contents of steps S101 to S104, in this application, first, S target object images are obtained, where S is an integer greater than or equal to 1. Then, based on the target prompt word, target semantic feature information of the S target object images is determined, wherein the target prompt word includes a question text that distinguishes objects of different categories, and the target semantic feature information is used to represent the answer information of each target object image based on the question text in the target prompt word. Then, the target semantic feature information of each target object image is input into the target model, and a label of each target object image is determined based on the target model, wherein the label of the target object image is used to describe the type information of the target object in the target object image, and the target model is a classification model obtained by training an initial model based on the semantic feature information of images of different categories. Finally, a target annotated image set is determined based on the label of each target object image. That is, by integrating the fine-grained feature extraction guided by the target prompt word and the classification model training method, the purpose of achieving accurate image annotation in a fine-grained small sample data set is achieved, thereby achieving the technical effect of improving the accuracy of automatic annotation, especially effectively annotating fine-grained categories on a small sample data set, thereby solving the technical problem of insufficient accuracy of automatic annotation on a fine-grained small sample data set in the prior art.

[0048] In an optional embodiment, the target model is trained through the following steps: the intelligent labeling system first obtains an object image set and a target list, wherein the object image set includes M object images, and the question instructions included in the target list are used to distinguish the characteristics of objects of different categories, wherein M is an integer greater than 1; then, a target prompt word is constructed based on the question instruction in the target list, and then the initial model is iteratively trained according to the object image set and the target prompt word to obtain the target model.

[0049] Optionally, the object image set includes a dataset of M object images, where M is an integer greater than 1, indicating that the set contains at least two images. These images may be from different scenes, such as inside a bank or in a self-service area, but all contain target objects that require fine-grained classification.

[0050] Optionally, a target list contains questions designed to distinguish features of different object categories, guiding the multimodal large model to understand and extract semantic feature information about the target objects. For example, a target list might include questions such as "Is the target wearing a uniform?" and "Is the target carrying anything?" to help distinguish between bank employees, customers, and security guards.

[0051] Optionally, in this embodiment, the initial model refers to a GBDT model that has not undergone specific training. It is a basic model with certain classification capabilities, but requires training on a specific dataset to optimize its performance so that it can accurately distinguish different fine-grained categories in financial scenarios.

[0052] Optionally, before training the target model, the intelligent labeling system needs to collect a set of object images (i.e., a set of object images), and the target objects in this set of images will be used for subsequent model training. At the same time, the system also needs to determine a target list, which contains multiple question instructions, each of which targets a certain feature and aims to distinguish different categories of target objects. Then, the intelligent labeling system will construct a target prompt word for input into the multimodal large model based on the question instructions in the target list. This prompt word usually contains instructions for the model, such as "Please answer the characteristics of the target object based on the following questions", as well as a specific list of question instructions. The process of constructing target prompt words is key to ensuring that the model can correctly extract semantic feature information from the image. The design of the prompt word needs to take into account the way the question is expressed, ensure that it can be understood by the model, and guide the model to generate accurate feature descriptions.

[0053] Optionally, the intelligent annotation system will then iteratively train the initial model. First, the system manually annotates all or a small portion of the object image collection. These images and their corresponding target semantic features constitute the training data. The intelligent annotation system then feeds this training data into the initial model. Through multiple iterations of training, the model's parameters are adjusted to optimize its classification performance, ultimately yielding the target model.

[0054] As can be seen from the above, the intelligent tagging system utilizes a limited set of object images and a list of targets. By constructing target cues and conducting iterative training, it develops a target model capable of effective fine-grained classification. The design of target cues ensures that the model understands and extracts the semantic features of objects in images, rather than relying solely on visual features. This enhances the model's generalization and ability to distinguish fine-grained categories. Through multiple training iterations, the intelligent tagging system enables the target model to continuously learn and adjust, optimizing its classification decisions and ultimately achieving high classification accuracy.

[0055] In an optional embodiment, the intelligent labeling system obtains N images, where N is an integer greater than or equal to 1, and then detects objects in each image through a first model, and intercepts the image area where the object is located in each image, wherein the first model is used to identify the object in the image, and then forms an object image set according to the image area corresponding to the object in each image.

[0056] Optionally, the first model is a pre-trained YOLO model (an algorithm for real-time object detection), which is used for preliminary detection of objects in an image. The first model can identify multiple objects in an image and provide a detection box for each object, that is, the location range of the object in the image. It is fast and easy to use.

[0057] Optionally, the intelligent annotation system collects N images from various sources. These images may come from surveillance cameras within the bank, in self-service banking areas, at customer service centers, or from on-site footage of specific events. The number of collected images, N, is any integer greater than or equal to 1, allowing the system to flexibly adapt to datasets of varying sizes.

[0058] Optionally, the image region refers to the portion of the image where each object detected by the first model is located, that is, the image within the detection box.

[0059] Optionally, after acquiring N images, the intelligent labeling system will use the first model (YOLO) to process each image and identify the target object. The YOLO model can process the entire image at once and directly predict the location and category of the object. Although in this system, it is mainly used for object location detection, and category identification will be completed through more precise mechanisms in subsequent steps. The YOLO model will output the coordinate information of each detected object, that is, the coordinates of the upper left and lower right corners of the detection box, which constitutes the preliminary positioning of the object position.

[0060] Optionally, based on the detection frame coordinates provided by the YOLO model, the intelligent annotation system automatically captures the image region where each target object is located, extracting the portion within each detection frame from the original image. This step separates each target object from the background, facilitating more detailed feature analysis and classification in subsequent steps. Each captured image region is associated with the original image and the detection frame information provided by YOLO, ensuring that subsequent processing can accurately map the classification results to specific locations in the original image. Finally, the intelligent annotation system aggregates all captured object image regions into a single object image collection.

[0061] From the above content, we can see that the intelligent labeling system can automatically process any number of images, and has achieved a high degree of automation from preliminary detection to extraction of target objects, greatly improving the efficiency of data preprocessing; the use of the YOLO model can quickly and accurately identify objects in each image, and locate the object position through the detection frame, preparing the image area for subsequent feature analysis; the captured image area constitutes an object image set, which provides a unified format and easy-to-operate data set for the subsequent processing of large multimodal models and GBDT classifiers, facilitating the training and reasoning of fine-grained classification.

[0062] In an optional embodiment, the intelligent labeling system inputs the object image set and the target prompt word into a multimodal large model, generates semantic feature information of each object image in the object image set based on the multimodal large model, then converts the semantic feature information of each object image in the object image set into a semantic feature list, and constructs a semantic feature list set based on the semantic feature list of each object image, wherein the information in each semantic feature list corresponds one-to-one to the question instruction in the target list, and then iteratively trains the initial model based on the object image set, the semantic feature list set and the metadata features to obtain the target model, wherein the metadata features are used to represent the temporal information and spatial information related to the image.

[0063] Optionally, semantic feature list: organizes the semantic feature information generated by the multimodal large model into a structured list format for further processing and analysis.

[0064] Optionally, metadata features: other information besides the image content, such as shooting time, shooting location, image size, etc., are used to characterize additional attributes of the image in time and space, which helps the model understand the context of the image.

[0065] Alternatively, the intelligent tagging system takes a pre-prepared set of object images and a pre-constructed target prompt as input to a large multimodal model. The model then analyzes each image in the set based on the question provided by the target prompt, generating semantic feature information. This information is a textual description of the characteristics of the target object in the image, such as "the target is wearing a uniform" or "the target is carrying a backpack."

[0066] Optionally, the intelligent tagging system then converts the semantic feature information for each image generated by the multimodal large model into a structured list format, with each item in each list corresponding to a question instruction in the target list. This conversion ensures the standardization of feature information, facilitating subsequent statistical analysis and model training. Once the conversion is complete, the system will obtain a set of semantic feature lists, which contains the semantic feature information of all images in the object image collection, organized in a list format.

[0067] Optionally, the intelligent labeling system can demonstrate strong adaptability in different scenarios through the image understanding and information extraction capabilities of a large multimodal model, and can maintain high performance regardless of changes in lighting conditions, target categories, or shooting angles.

[0068] Optionally, the intelligent labeling system combines the detection capabilities of YOLO, the image information extraction capabilities of multimodal large models, and the classification capabilities of GBDT. It can maintain stable performance under extremely small sample conditions and avoid overfitting problems caused by insufficient data.

[0069] Optionally, the intelligent tagging system uses a collection of object images, a collection of semantic feature lists, and metadata features as input for iterative training of the initial model. The intelligent tagging system can select a small amount of data from the object collection as training samples to train the initial model. During training, the initial model gradually learns how to identify and distinguish different categories of target objects based on the semantic feature lists and metadata features. Through multiple rounds of iteration, the model's classification accuracy continuously improves until it reaches a predetermined performance standard, resulting in an optimized target model.

[0070] Alternatively, the intelligent tagging system effectively improves fine-grained classification capabilities by combining the semantic information extraction capabilities of a large multimodal model with the precise classification capabilities of GBDT. Furthermore, by extracting semantic features of the target rather than general image features, the intelligent tagging system is applicable to most scenarios. Accurate classification can be achieved as long as the target is logically and semantically similar to a small sample, without requiring similarity in specific image parameters (such as camera angle, lighting conditions, clarity, resolution, etc.).

[0071] As can be seen from the above, the intelligent tagging system integrates a collection of object images, target cues, a list of semantic features, and metadata features to efficiently iteratively train the initial model, ultimately obtaining a target model capable of accurate, fine-grained classification. This system leverages not only the visual features of the image but also semantic and metadata features, enabling the model to understand the target object from multiple dimensions and improving classification accuracy. The inclusion of metadata features, particularly temporal and spatial information, enables the model to consider the context of the target object, which is particularly important for internal control and compliance analysis in financial scenarios, helping to improve the consistency and stability of classification decisions.

[0072] In an optional embodiment, the intelligent labeling system first determines the target model parameters of the initial model, and updates the initial model to a second model based on the target model parameters, wherein the target model parameters are model hyperparameters determined based on a cross-validation method, and then obtains a second set, wherein the second set includes the actual category of each object image in the object image set, and then constructs a training data set based on the second set, the semantic feature list set, and the metadata features, wherein the second set is used as a training label set, and the semantic feature list set and the metadata features are used as a training sample set, and finally the second model is iteratively trained according to the training data set until the number of iterations of the second model is greater than a preset number or the training error of the second model is lower than a set threshold, thereby obtaining a target model.

[0073] Optionally, the intelligent labeling system first uses cross-validation technology to conduct a series of experiments on the initial model to test the model performance under different hyperparameter configurations (such as tree depth, regularization coefficient, etc.). By comparing the model performance on differently divided data sets, the system can determine a set of optimal target model parameters. Next, the intelligent labeling system uses this set of target model parameters to update the initial model and optimize it into a second model. This process is essentially to initialize the model parameters to ensure that it has a good performance foundation in the early stages of network training.

[0074] Optionally, before training the second model, the system needs to obtain a second set of ground-truth class labels. This step typically involves manual labeling, which, while relatively labor-intensive (since only a small number of samples need to be labeled), is crucial for improving model accuracy. The ground-truth classes are typically annotated by experts or domain experts based on their in-depth understanding of the images, ensuring the accuracy and reliability of the second set and providing high-quality training labels for model training.

[0075] Optionally, the intelligent tagging system combines the second set (containing actual category information), the semantic feature list set, and metadata features (such as the location of the detection box and the time the image was taken) to construct a comprehensive training dataset. In this dataset, the semantic feature description and metadata features of each object instance are clearly labeled with its actual fine-grained category, forming the input and output of model training. This process ensures the completeness and accuracy of the training dataset, providing a rich information resource for the model's deep learning, helping the model better understand and distinguish different fine-grained categories.

[0076] Optionally, the intelligent labeling system will use the constructed training data set to iteratively train the second model. In each iteration, the model will adjust its parameters based on the training samples and the corresponding training labels in order to reduce prediction errors and improve classification accuracy. Iterative training will continue until one of two conditions is met: the number of iterations of the second model exceeds the pre-set upper limit, or the training error of the second model falls below the set minimum threshold. The purpose of this is to ensure that the model has fully learned the information of the data set without overfitting and maintaining good generalization capabilities. When the iterative training is completed, the second model is optimized to the final target model, which should be able to effectively perform fine-grained classification of new object images based on the semantic feature list and metadata features, and the classification accuracy is higher than that of the initial model.

[0077] From the above content, it can be seen that the model parameters determined by the intelligent labeling system through cross-validation ensure that the second model has a good performance foundation when training begins, reducing the risk of blind training; the training data set constructed by combining the second set of manually annotated, the semantic feature list set, and the metadata features provides a rich and accurate learning resource for the model; setting the upper limit of the number of iterations and the training error threshold can balance the learning effect and computational cost of the model, avoid overtraining, and ensure that the model achieves the expected classification performance within a reasonable time. Through the above steps, the intelligent labeling system not only improves the classification accuracy of the model, but also enhances the robustness and automation of the model under small sample data conditions, providing an efficient and reliable technical solution for fine-grained image classification in financial internal control and compliance management.

[0078] In an optional embodiment, each iterative training includes the following steps: the intelligent labeling system first predicts the training sample set through the second model to obtain a first prediction result corresponding to each training sample in the training sample set, then determines the error value between the first prediction result corresponding to each training sample and the training label corresponding to each training sample, and then constructs a target decision tree based on the error value between the first prediction result corresponding to each training sample and the training label corresponding to each training sample, wherein the target decision tree is used to correct the error value, and finally updates the second model based on the target decision tree.

[0079] Optionally, at the beginning of each iterative training, the intelligent labeling system will use the current version of the second model to make predictions on the training sample set. The purpose of this step is to evaluate the predictive performance of the model in its current state, that is, its ability to classify the training data. The intelligent labeling system will then compare the first prediction result of each training sample with the corresponding training label and calculate the difference between the two, that is, the error value. The calculation of the error value is the key to evaluating the accuracy of the model's prediction and is also the basis for subsequent model optimization. Based on the calculated error value, the intelligent labeling system will construct a target decision tree. This decision tree will reduce the prediction error of those samples that were predicted incorrectly by learning new feature segmentation points and classification rules, thereby improving the overall classification performance of the model.

[0080] Optionally, the construction of the target decision tree is the core of the GBDT algorithm. It uses an additive strategy, adding a new decision tree at each iteration to gradually correct the model's prediction error. The new tree will focus on learning samples that were previously predicted inaccurately, thereby improving the model's classification ability on difficult samples.

[0081] Optionally, the intelligent labeling system will merge the newly constructed target decision tree into the current second model in a final iteration step to perform a model update. This update process ensures that the model gradually learns and corrects errors. As the number of iterations increases, the model's classification performance gradually improves until it reaches the predetermined training standard or performance metric. Updating the second model is the ultimate goal of iterative training. By continuously adding and correcting error-prone decision trees, the model learns increasingly complex classification patterns, thereby demonstrating greater accuracy and robustness on fine-grained classification tasks.

[0082] From the above, we can see that the model can construct a new decision tree based on the error of each prediction, gradually correct errors, and improve classification performance; the target decision tree constructed in each iteration will focus on those samples with incorrect predictions, ensuring that the model can also be improved on the most difficult-to-classify samples, thereby enhancing the model's generalization ability; by continuously adding decision trees that correct errors, the model's classification ability can be gradually improved, ultimately achieving a high classification accuracy rate. This iterative training method conducts targeted learning on the model's errors during the prediction process, effectively improving the model's classification accuracy and robustness, especially when dealing with fine-grained classification tasks, which can significantly improve the model's performance in distinguishing complex categories. In this way, the intelligent labeling system can train a model that can accurately perform fine-grained classification with limited training samples and computing resources, thereby greatly improving the efficiency and quality of automated labeling of financial internal control compliance data, and providing strong support for subsequent data analysis, model training and other tasks.

[0083] In an optional embodiment, the second model is updated based on the target decision tree, including: the intelligent labeling system trains the target decision tree based on the error value between the first prediction result corresponding to each training sample and the training label corresponding to each training sample, and then predicts the training samples through the trained target decision tree to obtain the second prediction result corresponding to each training sample, and then merges the first prediction result and the second prediction result corresponding to each training sample to obtain the target prediction result corresponding to each training sample, and finally updates the second model based on the target prediction result.

[0084] Optionally, the intelligent labeling system calculates the error between the target model's first prediction for each training sample and the actual training label manually annotated for that sample. These errors reflect the inaccuracy of the current model's classification and are a key metric for model optimization. The system then uses these errors as feedback to train the target decision tree. Through an iterative learning process, the tree's structure and parameters are adjusted to enable it to more accurately predict the class of training samples and reduce classification error.

[0085] Optionally, after the target decision tree is trained, the intelligent labeling system will use the optimized decision tree again to predict the training samples and generate a second prediction result. This process helps verify the learning effect of the target decision tree, that is, to observe whether the trained decision tree can more accurately identify the target object category in the sample. The system merges the first and second prediction results to generate a target prediction result for each training sample. This merger may adopt the majority voting principle, that is, the category with the most votes in the first and second prediction results is used as the target prediction result; or it may adopt a weighted average method, giving different prediction results different weights based on their performance during the training process to obtain more robust classification judgments. The target prediction results obtained after the merger will be used as feedback, and the intelligent labeling system will adjust the parameters of the second model based on these results. This process can be completed using gradient descent, random forest optimization, or other update strategies suitable for the GBDT model to ensure that the model can perform fine-grained classification more accurately.

[0086] From the above content, we can see that the intelligent labeling system evaluates the prediction error of each training sample and uses these errors to train the target decision tree to ensure that the model can correct its classification strategy in a targeted manner and gradually reduce errors; the preliminary prediction results and the optimized prediction results are combined to generate a more reliable target prediction result. This method increases the stability of the model prediction and prevents the uncertainty caused by over-reliance on a certain prediction; based on the target prediction result, the system can effectively update the second model, improve the overall classification performance of the model, and make the model more accurate and efficient when processing new data.

[0087] In an optional embodiment, Figure 2 is a schematic diagram of an optional image annotation method according to an embodiment of the present application, such as Figure 2 As shown, it mainly includes the following steps:

[0088] (1) First, obtain some pictures.

[0089] (2) Then, the YOLO model is used to detect coarse-grained targets in the image. The specific process is as follows: the pre-trained YOLO model is used to detect coarse-grained classification targets (such as all "people") in the input image. The result of this step is multiple detection frames corresponding to each image. Each detection frame is in the form of [x1, y1, x2, y2], which means the coordinates of the upper left corner and the lower right corner. Then, for each detection frame of each image, the part of the image framed by the detection frame is cut out according to the coordinates of the detection frame. This part of the screenshot is used as the input of the subsequent large model. At the same time, the correspondence between each screenshot and the original image and the original detection frame is recorded, so that the fine-grained classification of each screenshot can be corresponded back to the original image and the original detection frame in the final output result. At the same time, the large-scale model receives a list of questions based on the fine-grained classification characteristics of the target, obtained through observation and experience from analyzing the image. These questions cover a variety of fine-grained classification characteristics (for example, when classifying "bank employee," "customer," and "security guard," these questions include, but are not limited to, "Is the target wearing a uniform?", "Is the target wearing a white shirt?", "Is the target wearing a helmet?", and "Is the target carrying personal items such as a backpack or handbag?"). In this step, as many questions as possible can be prepared, based on common sense, relevant domain knowledge, or observations of the dataset, as long as they are relevant to the characteristics of the classification target. Questions that are not helpful for fine-grained classification will be automatically used less frequently or not at all in the subsequent model. All the questions generated are then numbered and organized into a string. Auxiliary instructions are added, such as "This is an image. Please answer the following questions in order based on the image content" and "For each question, please simply answer "yes" without punctuation or additional explanation." This string serves as a prompt for the large-scale model in the next step.

[0090] (3) Input the prompt words and screenshots obtained above into the multimodal large model. Utilizing the multimodal large model's ability to understand and analyze images and text, the large model outputs the semantic features of the target corresponding to each image in turn according to each question in the prompt words (features that can be understood by humans and expressed in language, such as: the target is wearing a white shirt / the target is not wearing a white shirt, which is different from general features obtained by calculating based on image pixels, such as frequency domain features obtained by Fourier transform, edge features obtained by edge detection algorithm, etc.). The result of this step is a string corresponding to each screenshot, which contains a sequence number and the answer.

[0091] (4) For each string segment output by the large model, post-process it through the code (for example, split the string by line breaks, and then remove the numerical sequence and colon respectively) to obtain a list of semantic features. The length of the list is consistent with the length of the large model question list mentioned above, and each semantic feature corresponds to a question.

[0092] (5) The screenshots obtained in (2) above need to be labeled with categories. Select a small number of data samples (select some screenshots), and manually label each detection box of each picture in the small sample, that is, the corresponding screenshot obtained in step 2, with fine-grained categories, for example, "people" are divided into "bank staff", "customers", "security guards", etc. Depending on the specific situation of the data set and the specific differences in the fine-grained categories, the number of samples required is also different. Usually, 1-3 pictures are required for each fine-grained category in each scene. This number is greatly reduced compared to pure manual labeling or existing small sample learning methods. The labels obtained in this step correspond to the screenshots obtained in step (2) and the semantic feature list obtained in step (4).

[0093] (6) The obtained small sample manual annotations are used as fine-grained labels (each type of annotated image corresponds to approximately three images), and the corresponding list of semantic features obtained in (4) and other related metadata (such as the time the image was taken, the position and size of the detection frame, etc.) are used as features to obtain a well-annotated small sample data set, and combined with the GBDT classification algorithm to train the GBDT classifier. Among them, the optimal training parameters of the model can be automatically obtained through grid search cross-validation. In addition, GBDT can automatically plan the use of features, including order, frequency, importance, etc. Secondary features will be used less or not at all, so it has good invalid redundancy for the problem list design in (2). This step of training will obtain a GBDT classifier, whose input is semantic features and output is fine-grained category labels.

[0094] (7) Use the trained GBDT classifier to automatically classify the unlabeled images in the screenshots obtained in step (2). The input of the classifier is the list of semantic features corresponding to the unlabeled screenshots, and the output is the corresponding fine-grained category label, which corresponds to the original image and the original detection box.

[0095] (8) After the above steps, an image dataset is obtained, in which each image has one or more corresponding detection boxes, and each detection box has a corresponding fine-grained category label. This dataset can be further used for data analysis and other model training. For example, this dataset can be used for YOLO training to obtain a real-time detection model that does not rely on a large multimodal model.

[0096] The present application also provides an image annotation device. It should be noted that the image annotation device of the present application can be used to execute the image annotation method provided in the present application. The image annotation device provided in the present application is introduced below.

[0097] According to an embodiment of the present application, a device for implementing the above-mentioned image annotation method is also provided. Figure 3is a schematic diagram of an optional image annotation device according to an embodiment of the present application, such as Figure 3 As shown, the apparatus includes: a first acquiring unit 301 , a first determining unit 302 , a second determining unit 303 and a third determining unit 304 .

[0098] Optionally, the first acquisition unit 301 is used to acquire S target object images, where S is an integer greater than or equal to 1; the first determination unit 302 is used to determine the target semantic feature information of the S target object images based on the target prompt word, wherein the target prompt word includes a question text for distinguishing objects of different categories, and the target semantic feature information is used to characterize the answer information of each target object image based on the question text in the target prompt word; the second determination unit 303 is used to input the target semantic feature information of each target object image into the target model, and determine the label of each target object image based on the target model, wherein the label of the target object image is used to describe the type information of the target object in the target object image, and the target model is a classification model obtained by training the initial model based on the semantic feature information of images of different categories; the third determination unit 304 is used to determine the target annotated image set based on the label of each target object image.

[0099] Optionally, the second determination unit 303 includes: a first acquisition subunit, a first construction subunit, and a first training subunit. The first acquisition subunit is configured to acquire an object image set and a target list, wherein the object image set includes M object images, and the question instructions included in the target list are used to distinguish the features of objects of different categories, where M is an integer greater than 1; the first construction subunit is configured to construct a target prompt word based on the question instruction in the target list; and the first training subunit is configured to iteratively train the initial model based on the object image set and the target prompt word to obtain a target model.

[0100] Optionally, the first acquisition subunit includes: a first acquisition module, a first interception module, and a first composition module. The first acquisition module is configured to acquire N images, where N is an integer greater than or equal to 1; the first interception module is configured to detect objects in each image using a first model and intercept the image region containing the object in each image, wherein the first model is configured to identify objects in the image; and the first composition module is configured to assemble an object image set based on the image region corresponding to the object in each image.

[0101] Optionally, the first training subunit includes: a first generation module, a first conversion module, and a first training module. The first generation module is used to input the object image set and the target prompt word into the multimodal large model, and generate semantic feature information for each object image in the object image set based on the multimodal large model; the first conversion module is used to convert the semantic feature information of each object image in the object image set into a semantic feature list, and form a semantic feature list set based on the semantic feature list of each object image, wherein the information in each semantic feature list corresponds one-to-one to the question instruction in the target list; the first training module is used to iteratively train the initial model based on the object image set, the semantic feature list set, and the metadata features to obtain the target model, wherein the metadata features are used to represent the temporal and spatial information related to the image.

[0102] Optionally, the first training module includes: a first determination submodule, a first acquisition submodule, a first construction submodule, and a first training submodule. The first determination submodule is used to determine the target model parameters of the initial model and update the initial model to the second model based on the target model parameters, wherein the target model parameters are model hyperparameters determined based on a cross-validation method; the first acquisition submodule is used to obtain the second set, wherein the second set includes the actual category of each object image in the object image set; the first construction submodule is used to construct a training data set based on the second set, the semantic feature list set, and the metadata features, wherein the second set is used as the training label set, and the semantic feature list set and the metadata features are used as the training sample set; the first training submodule is used to iteratively train the second model according to the training data set until the number of iterations of the second model is greater than a preset number or the training error of the second model is lower than a set threshold, thereby obtaining the target model.

[0103] Optionally, the first training submodule includes: a first prediction component, a first determination component, a first construction component, and a first update component. The first prediction component is used to predict the training sample set using the second model to obtain a first prediction result corresponding to each training sample in the training sample set; the first determination component is used to determine the error value between the first prediction result corresponding to each training sample and the training label corresponding to each training sample; the first construction component is used to construct a target decision tree based on the error value between the first prediction result corresponding to each training sample and the training label corresponding to each training sample, wherein the target decision tree is used to correct the error value; and the first update component is used to update the second model based on the target decision tree.

[0104] Optionally, the first update component includes: a first training subcomponent, a first prediction subcomponent, a first merging subcomponent, and a first update subcomponent. The first training subcomponent is used to train the target decision tree based on the error between the first prediction result corresponding to each training sample and the training label corresponding to each training sample; the first prediction subcomponent is used to predict the training samples using the trained target decision tree to obtain a second prediction result corresponding to each training sample; the first merging subcomponent is used to merge the first prediction result and the second prediction result corresponding to each training sample to obtain a target prediction result corresponding to each training sample; and the first update subcomponent is used to update the second model based on the target prediction result.

[0105] According to another aspect of the present application, a computer-readable storage medium is provided. The computer-readable storage medium includes a stored executable program, wherein when the executable program runs, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned image annotation method.

[0106] According to another aspect of the present application, an electronic device is also provided, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned image annotation method.

[0107] According to another aspect of an embodiment of the present application, a computer program product is further provided, including computer instructions, which implement the steps of the above-mentioned image annotation method when executed by a processor.

[0108] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0109] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0110] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0111] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0112] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0113] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0114] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image annotation method, characterized in that: include: Acquire S target object images, where S is an integer greater than or equal to 1; Determining target semantic feature information of S target object images based on a target prompt word, wherein the target prompt word includes question text for distinguishing objects of different categories, and the target semantic feature information is used to represent answer information of each target object image based on the question text in the target prompt word; Inputting target semantic feature information of each target object image into a target model, and determining a label for each target object image based on the target model, wherein the label of the target object image is used to describe type information of the target object in the target object image, and the target model is a classification model obtained by training an initial model based on semantic feature information of images of different categories; A target annotated image set is determined according to the label of each target object image.

2. The image annotation method according to claim 1, characterized in that: The target model is trained by the following steps: Obtain an object image set and a target list, wherein the object image set includes M object images, and the question instructions included in the target list are used to distinguish features of objects of different categories, wherein M is an integer greater than 1; constructing the target prompt word based on the question instruction in the target list; The initial model is iteratively trained according to the object image set and the target prompt word to obtain the target model.

3. The image annotation method according to claim 2, characterized in that: Get a collection of object images, including: Acquire N images, where N is an integer greater than or equal to 1; Detecting an object in each of the images using a first model, and intercepting an image region where the object in each of the images is located, wherein the first model is used to identify the object in the image; The object image set is formed according to the image area corresponding to the object in each of the images.

4. The image annotation method according to claim 2, characterized in that: Iteratively training the initial model according to the object image set and the target prompt word to obtain the target model includes: Inputting the object image set and the target prompt word into a multimodal large model, and generating semantic feature information of each object image in the object image set according to the multimodal large model; Converting the semantic feature information of each object image in the object image set into a semantic feature list, and forming a semantic feature list set based on the semantic feature list of each object image, wherein the information in each semantic feature list corresponds one-to-one to the question instruction in the target list; The initial model is iteratively trained according to the object image set, the semantic feature list set, and metadata features to obtain the target model, wherein the metadata features are used to represent temporal information and spatial information related to the image.

5. The image annotation method according to claim 4, characterized in that: Iteratively training the initial model according to the object image set, the semantic feature list set, and metadata features to obtain the target model includes: Determining target model parameters of the initial model, and updating the initial model to a second model according to the target model parameters, wherein the target model parameters are model hyperparameters determined based on a cross-validation approach; Acquire a second set, wherein the second set includes the actual category of each object image in the object image set; Constructing a training data set according to the second set, the semantic feature list set, and the metadata features, wherein the second set is used as a training label set, and the semantic feature list set and the metadata features are used as a training sample set; The second model is iteratively trained according to the training data set until the number of iterations of the second model is greater than a preset number or the training error of the second model is lower than a set threshold, thereby obtaining the target model.

6. The image annotation method according to claim 5, characterized in that: Each training iteration consists of the following steps: Predicting the training sample set using the second model to obtain a first prediction result corresponding to each training sample in the training sample set; Determine an error value between a first prediction result corresponding to each of the training samples and a training label corresponding to each of the training samples; Constructing a target decision tree according to an error value between the first prediction result corresponding to each of the training samples and the training label corresponding to each of the training samples, wherein the target decision tree is used to correct the error value; The second model is updated based on the target decision tree.

7. The image annotation method according to claim 6, characterized in that: Updating the second model based on the target decision tree includes: Training the target decision tree based on an error value between the first prediction result corresponding to each of the training samples and the training label corresponding to each of the training samples; Predicting the training samples using the trained target decision tree to obtain a second prediction result corresponding to each training sample; Merging the first prediction result and the second prediction result corresponding to each of the training samples to obtain a target prediction result corresponding to each of the training samples; The second model is updated based on the target prediction result.

8. An image annotation device, characterized in that: include: A first acquisition unit is configured to acquire S target object images, where S is an integer greater than or equal to 1; a first determining unit, configured to determine target semantic feature information of the S target object images based on a target prompt word, wherein the target prompt word includes question text for distinguishing objects of different categories, and the target semantic feature information is used to represent answer information of each target object image based on the question text in the target prompt word; a second determining unit, configured to input target semantic feature information of each target object image into a target model, and determine a label for each target object image based on the target model, wherein the label of the target object image is used to describe type information of the target object in the target object image, and the target model is a classification model obtained by training an initial model based on semantic feature information of images of different categories; The third determining unit is configured to determine a target annotated image set according to a label of each target object image.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located executes the image annotation method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: The method comprises one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the image annotation method according to any one of claims 1 to 7.

11. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the image annotation method according to any one of claims 1 to 7 are implemented.

Images