Method and device for determining target image set, electronic equipment and storage medium

Abstract

The invention discloses a target image set determination method and device, electronic equipment and a storage medium, and the method comprises the steps: determining a target object according to an original image, and determining a candidate attribute set matched with the target object; according to the attribute frequency and the co-occurrence frequency of each candidate attribute in the candidate attribute set, determining the attribute weight of each candidate attribute; according to the attribute weight of each candidate attribute in the candidate attribute set, determining a target attribute in each candidate attribute as an image tag matched with the target object; and determining a target image set according to the target object image corresponding to the target image tag. According to the invention, automatic, efficient and accurate image label determination can be realized, so that generation of an image training set or a test set is realized based on the image labels.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a target image set determination method and device, electronic equipment and a storage medium. BACKGROUND

[0002] With the continuous development of artificial intelligence models, image-text models have more and more extensive applications in content creation, intelligent interaction, cross-modal retrieval and visualization analysis.

[0003] In the training and testing process of an image-text matching model, constructing a training set and a test set of image-text matching descriptions with rich scenes, single attributes and multi-attribute combinations is crucial for model optimization. Existing public training sets and test sets are either single-scene or single-attribute. To obtain a training set and a test set of image-text matching descriptions with rich scenes, single attributes and multi-attribute combinations, the target objects of images are usually manually labeled, randomly or pre-set combined according to attribute classification, and then the target objects are labeled and classified by the model. Finally, the image-text matching is checked by a human, i.e., whether the target object and the attribute label match, and whether the attribute classification is correct. This method has high labor cost and low efficiency. SUMMARY

[0004] The present application provides a target image set determination method, device, electronic equipment and storage medium to realize automatic, efficient and accurate image label determination, and thus generate an image training set or test set based on image labels.

[0005] In a first aspect, the present application provides a target image set determination method, which comprises:

[0006] determining a target object from an original image and determining a candidate attribute set matched with the target object;

[0007] determining an attribute weight of each candidate attribute in the candidate attribute set according to an attribute frequency and a co-occurrence frequency of each candidate attribute;

[0008] determining a target attribute from each candidate attribute in the candidate attribute set according to the attribute weight of each candidate attribute in the candidate attribute set, as an image label matched with the target object;

[0009] determining a target image set according to a target object image corresponding to the target image label.

[0010] In a second aspect, the present application also provides a target image set determination device, which comprises:

[0011] a candidate attribute set determination module configured to determine a target object from an original image and determine a candidate attribute set matched with the target object;

[0012] The attribute weight determination module is configured to determine an attribute weight of each candidate attribute according to the attribute frequency and the co-occurrence frequency of each candidate attribute in the candidate attribute set.

[0013] The image attribute determination module is configured to determine a target attribute from the candidate attributes as an image label matched with the target object according to the attribute weight of each candidate attribute in the candidate attribute set.

[0014] The target image set determination module is configured to determine a target image set according to the target object image corresponding to the target image label.

[0015] In a third aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method for determining a target image set when executing the program.

[0016] In a fourth aspect, a storage medium storing computer executable instructions is provided, and the computer executable instructions are used to execute the method for determining a target image set when executed by a computer processor.

[0017] The technical solution of the embodiments of the present application determines a target object in an original image, determines a candidate attribute set for the target object, determines the attribute frequency and the co-occurrence frequency of each candidate attribute in the candidate attribute set, determines the attribute weight of each candidate attribute based on the attribute frequency and the co-occurrence frequency, filters each candidate attribute in the candidate attribute set according to the attribute weight to obtain a target attribute as an image label of the target object, and finally determines a target image set based on the target object image corresponding to the target image label. The technical solution of the embodiments of the present application realizes automatic labeling of the attributes of the target object in the image, reduces the labor cost of image labeling and classification, and can realize fast image attribute classification and construction of a training set and / or a test set of the same image attribute according to the image attributes of each target object in the original image.

[0018] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0020] Figure 1 is a flow chart of a target image set determination method provided by an embodiment of the present application;

[0021] Figure 2 is a schematic diagram of a dynamic tree provided by an embodiment of the present application;

[0022] Figure 3 is a flow chart of a target image set determination method provided by an embodiment of the present application;

[0023] Figure 4 is a structural schematic diagram of a target image set determination device provided by an embodiment of the present application;

[0024] Figure 5 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the personnel in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. In the embodiments of the present application, some software, components, models and the like may be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the implementation of the technical solutions of the present application, but does not mean that the applicant has or will necessarily use the scheme.

[0027] The acquisition, transmission, storage, use, processing and the like of data in the technical solutions of the present application all comply with the relevant provisions of national laws and regulations.

[0028] Embodiment one

[0029] Figure 1 A flowchart of a method for determining a target image set is provided for Embodiment One of the present application. This embodiment can be applied to automatic labeling of target object attributes of original images and construction of image training sets / test sets of target objects with the same label. The method can be executed by a target image set determination device, which can be implemented in the form of hardware and / or software and can be configured in a server.

[0030] As shown in Figure 1 , the method comprises:

[0031] S110, determining a target object according to an original image and determining a candidate attribute set matched with the target object.

[0032] The original image refers to an image that has not been labeled with target object attributes. The original image can be an image in a full-scene image set or an image in a specific-scene image set.

[0033] The target object can be a vehicle, a person, or other specific objects, etc. The target object can be in the form of a target object detection box obtained by target object recognition of the original image or in the form of an image obtained by cropping the target object.

[0034] In this embodiment, the target object can be determined according to the original image by using a target object detection algorithm based on deep learning or a target object detection algorithm based on computer vision, etc. For example, the original image can be recognized by using a DINO-X (Detection In Network-X) model to obtain the detection box of each target object in the original image, the corresponding target object type, and the position coordinates of the target object detection box in the original image.

[0035] The candidate attribute is used to represent the appearance, state, behavior, and relationship with other target objects, etc. of the target object. For example, the candidate attribute of a person as a target object can be whether to wear a hat (appearance), gender (appearance), moving or still (state), and whether to cross the road (behavior), etc. For example, the candidate attribute of a vehicle as a target object can be vehicle color (appearance), license plate color (appearance), license plate recognition result (appearance), lane (state), and speed (state), etc.

[0036] The candidate attribute can be a single attribute, i.e., an attribute that cannot be further split, and exemplary, taking a person as the target object, the single attribute can be long hair / short hair, wearing glasses, etc. The candidate attribute can also be a combined attribute, i.e., an attribute that can be further split, and exemplary, taking a person as the target object, the combined attribute can be a white T-shirt, which can be further split into a white (color) upper garment and an upper garment style (T-shirt).

[0037] The candidate attribute set includes at least two candidate attributes, and if only one candidate attribute is determined for the target object, the candidate attribute can be directly used as the image label of the target object. When multiple candidate attributes of the target object are determined, the technical solution of the embodiment is used to filter the target attribute from the candidate attributes as the image label of the target object.

[0038] In an optional embodiment, the candidate attribute set matched with the target object can be determined by pre-determining different candidate attribute types for different types of target objects, and performing attribute recognition on the types of candidate attributes respectively. Exemplarily, taking a person as the target object, the human body region can be divided based on the computer vision method to obtain an upper garment region and a lower garment region; edge detection is performed first to extract contour features and determine the style type of the upper garment or the lower garment; and then color feature extraction is performed to determine the color of the upper garment or the lower garment.

[0039] In another optional embodiment, the candidate attribute set matched with the target object can also be determined by a deep learning-based method to automatically learn the features of the target object region and output more complex and multi-dimensional attributes. Exemplarily, based on target object recognition of the original image by a target detection model, the target object can be obtained, an attribute prediction head is added to the target detection model, and multi-dimensional attributes are output at the same time as the target object. After the target object is obtained, the candidate attributes of the target object can also be obtained through an attribute recognition model. The attribute recognition model can be based on a CNN (Convolutional Neural Network) or a ViT (Vision Transformer) and learn the attribute labels of the target object through a classification task. Exemplarily, the attribute recognition model can use an InternVL3-8B model, which can perform detailed attribute description on the target object image and output in a fixed format, thereby laying a foundation for subsequent image attribute determination.

[0040] Further, S110 can further include:

[0041] S111, performing target object recognition on the original image to obtain the target object;

[0042] S112, attribute marking is performed on the target object to obtain an attribute dictionary of the target object;

[0043] S113, word segmentation matching is performed on each attribute in the attribute dictionary to obtain a candidate attribute set composed of at least two candidate attributes;

[0044] The candidate attribute is a single attribute and / or a combined attribute.

[0045] The embodiment provides a technical scheme of performing word segmentation matching on an attribute dictionary and obtaining at least two candidate attribute sets.

[0046] The attribute dictionary refers to a standardized definition of attributes of a target object through preset unified, standardized attribute categories, value ranges, and description rules. For example, taking a vehicle as the target object, the attribute dictionary can include: a color attribute with a value range of {red, blue, white, black, silver, and other colors}, a vehicle type attribute with a value range of {sedan, SUV, truck, bus, and motorcycle}, and a state attribute {still, driving, and stopping}. For the same type of target object, the attribute dictionary can be consistent.

[0047] It can be understood that the attribute dictionary obtained by performing attribute marking on the target object can be a multi-dimensional attribute, and therefore, word segmentation matching is performed on each attribute in the attribute dictionary, especially the multi-dimensional attribute. For example, taking a person as the target object, if the attribute dictionary includes a multi-dimensional attribute "white hat", the attribute includes both white (color) and clothing type (hat), and therefore, the attribute can be segmented into "white" and "hat". Since color, pattern, and the like belong to adjectives in the multi-dimensional attribute, the adjectives are not classified as attributes alone, and therefore, after segmentation, the subject alone or the adjective + subject can be obtained by re-matching, for example, "white hat" and "hat".

[0048] Further, the attribute after segmentation must be distinctive, for example, for the multi-dimensional attribute "brown umbrella", the attribute after segmentation can be "umbrella" and "brown umbrella", but not "umbrella". Specifically, non-distinctive words can be pre-counted and set, and the non-distinctive words are not split.

[0049] Further, when there are multiple adjectives, the adjectives can also be split and re-matched to obtain multiple recombined attributes. For example, for the multi-dimensional attribute "black plaid shirt", after word segmentation matching, different attributes such as "black shirt", "black plaid shirt", "plaid shirt", and "shirt" can be obtained.

[0050] In the embodiment, the tokenization matching is performed on each attribute in the attribute dictionary, especially the multi-dimensional attribute, and a plurality of candidate attribute sets can be obtained. Taking the above example as an example, if the other attributes are single attributes, two candidate attribute sets can also be obtained based on a target object, which are {“white hat”, “xx”, “xx”, …} and {“hat”, “xx”, “xx”, …}. The advantage of such setting is that for the same target object, a plurality of candidate attribute sets can be obtained, and when the image label is determined based on the attribute weight of each candidate attribute, the image label determined based on the plurality of candidate attribute sets can be used as the retrieval basis for constructing the training set and / or the test set.

[0051] In S120, the attribute weight of each candidate attribute is determined according to the attribute frequency and the co-occurrence frequency of each candidate attribute in the candidate attribute set.

[0052] The attribute frequency is used to represent the appearance frequency of the candidate attribute currently processed in all candidate attributes. The “all candidate attributes” here can be all target objects, all candidate attributes corresponding to each attribute category, or all candidate attributes of the attribute category to which the candidate attribute currently processed belongs.

[0053] The co-occurrence frequency is used to represent the frequency of simultaneous appearance between the candidate attribute currently processed and other candidate attributes. Since the values corresponding to the same attribute category have co-occurrence exclusivity, for example, a car model attribute cannot be a car and an SUV at the same time, the other candidate attributes can be all candidate attributes corresponding to other attribute categories.

[0054] Further, S120 can further include:

[0055] In S121, for each candidate attribute in the candidate attribute set, the attribute frequency of the candidate attribute is determined according to the number of times the candidate attribute appears in each candidate attribute corresponding to each target object and the total number of candidate attributes corresponding to the image of each target object.

[0056] In S122, the co-occurrence frequency matched with the candidate attribute is determined according to the attribute co-occurrence matrix, wherein the elements of the attribute co-occurrence matrix represent the probability of simultaneous appearance between different candidate attributes.

[0057] In S123, the attribute weight of the candidate attribute is determined according to the attribute frequency, the attribute frequency weight, the co-occurrence frequency of the candidate attribute, and the co-occurrence frequency weight.

[0058] The embodiment provides a specific calculation method of attribute frequency and co-occurrence frequency, and a specific implementation method of determining the attribute weight based on the attribute frequency and the co-occurrence frequency.

[0059] Specifically, a ratio of a number of occurrences of the candidate attribute currently processed in all candidate attributes to a total number of all candidate attributes is taken as the attribute frequency of the candidate attribute currently processed.

[0060] Taking the total number of all candidate attributes as K, a co-occurrence matrix C of KxK is constructed K×K For candidate attributes A i and A j , a number of simultaneous occurrences thereof is counted, and a ratio of the number of simultaneous occurrences of A i and A j to a total number of target object images is taken as a co-occurrence probability C i between A j and A ij in the co-occurrence matrix. A co-occurrence frequency matching the candidate attribute A i may be calculated by .

[0061] The attribute frequency weight and the co-occurrence frequency weight can be fixed values set in advance, or can be flexibly adjusted according to different scenes covered by the original image set. For example, the attribute frequency weight can be taken as 0.7, and the co-occurrence frequency weight can be taken as 0.3.

[0062] In an optional embodiment, S123 can include: taking a sum of a product of the attribute frequency of the candidate attribute and the attribute frequency weight and a product of the co-occurrence frequency of the candidate attribute and the co-occurrence frequency weight as the attribute weight of the candidate attribute.

[0063] In another optional embodiment, S123 can determine the attribute weight of the candidate attribute through the following formula: Wherein, A i represents the i-th candidate attribute, a represents the attribute frequency weight, β represents the co-occurrence frequency weight, F i represents the attribute frequency of the candidate attribute, C ij represents a co-occurrence probability of A i and A j in the attribute co-occurrence matrix, and K represents the number of candidate attributes.

[0064] In the embodiment, the attribute frequency and the co-occurrence frequency of the candidate attribute of the target object are calculated, and the attribute weight of the candidate attribute is determined based on the attribute frequency and the co-occurrence probability, so as to realize the quantitative representation of the scene correlation degree, the attribute correlation degree and the importance of the candidate attribute, thereby providing a data basis for the selection of the image label of the target object in the subsequent process.

[0065] S130, determining a target attribute from the candidate attributes according to the attribute weight of each candidate attribute in the candidate attribute set as the image label matched with the target object.

[0066] In the embodiment, the target attribute is filtered from the candidate attributes according to the attribute weight of each candidate attribute as the image label matched with the target object, which sets the attributes in the image label of the target object as high-frequency attributes, and the dimensions between the attributes in the image label are different, which not only realizes the automatic determination of the image label of the target object, saves the manual labeling cost, but also improves the accuracy and richness of the image label.

[0067] It should be noted that if the original image set is a fixed set of image quantity, the steps of S110-S130 can be performed once for each original image. If the original image set is a dynamic set in which original images are added or deleted, the attribute frequency and co-occurrence frequency of the candidate attributes in the candidate attribute set of each target object need to be recalculated, the attribute weight needs to be determined again, and the image label of each target object needs to be determined again every time an original image is added or deleted.

[0068] In an optional embodiment, S130 can include sorting the attribute weights of the candidate attributes in the candidate attribute set from large to small, and selecting the first predetermined number of candidate attributes as target attributes, i.e., image labels matched with the target object.

[0069] The predetermined number can be a fixed value, for example, 3; or it can be calculated according to the total number of candidate attributes in the candidate attribute set and a predetermined proportion, and then rounded up, for example, the predetermined proportion can be 50%, and if the candidate attribute set includes 5 candidate attributes, the calculated predetermined number is 3. Further, if the scene covered by the original image set is a single scene, the predetermined number or predetermined proportion corresponding to the scene is determined, and if the scene covered by the original image set is at least two scenes, different predetermined numbers or predetermined proportions can be set for different scenes.

[0070] In another optional embodiment, S130 can also include selecting, from the candidate attributes in the candidate attribute set, a candidate attribute with an attribute weight greater than or equal to a predetermined weight threshold as a target attribute, i.e., an image attribute matched with the target object.

[0071] Similarly, different weight thresholds can be set for different scenes covered by the original image set. Different weight thresholds can also be set for different attribute categories, and the candidate attributes of different attribute categories are respectively determined as target attributes according to their corresponding weight thresholds.

[0072] Further, if the number of target attributes obtained by using the implementation of the optional embodiment is less than or equal to the preset number threshold, the attribute weights of the candidate attributes other than the target attributes in the candidate attribute set can be sorted from large to small, and the first preset number of candidate attributes are taken as the target attributes. Similarly, the preset number can be a fixed value, or can be calculated according to the total number of candidate attributes in the candidate attribute set and the preset proportion. The advantage of such setting is that the multi-dimensionality of the image label of the target object can be ensured.

[0073] In yet another optional embodiment, after S120, a dynamic tree can also be constructed according to the candidate attribute set matched with each target object image; wherein the dynamic tree includes a root node and branches matched with each target object, and the child nodes in the branch matched with the target object are each candidate attribute in the candidate attribute set matched with the target object. S130 can also include: reordering each candidate attribute in the branch matched with the target object according to the attribute weight of each candidate attribute in the candidate attribute set; and taking the first preset number of layers of candidate attributes as the image attributes matched with the target object.

[0074] The embodiment provides a manner of realizing image attribute determination based on construction of a dynamic tree. Specifically, the root node corresponds to an original image set, and for each target object of each original image in the original image set, a branch is constructed according to each candidate attribute in the candidate attribute set corresponding to the target object, and each candidate attribute is taken as a child node in the branch. The initial ordering manner of each child node (candidate attribute) in a branch is not limited, and subsequent adjustment of the ordering according to the attribute weight of each candidate attribute is needed.

[0075] Figure 2 A schematic diagram of a dynamic tree is provided, Figure 2 Four branches are provided as examples in the embodiment, and each branch corresponds to a target object, and each child node in the branch constitutes a candidate attribute set of the target object corresponding to the branch.

[0076] Since each branch of the dynamic tree is reordered, the shallow candidate attributes of each branch are all high-frequency attributes, and taking the shallow candidate attributes as the image label ensures the reliability of the image label.

[0077] The embodiment realizes determination of the image label based on the dynamic tree. On one hand, when a new original image or target object is added, the new branch can be incorporated into the dynamic tree, and the image label can be quickly determined based on the dynamic tree structure. On the other hand, based on the dynamic tree structure, the target object or the original image can be conveniently searched or classified, and especially when the search condition or the classification condition is a combination attribute, the attribute information corresponding to the child nodes of each branch of the dynamic tree can be used to quickly determine the branch containing the combination attribute, and the image search or classification can be realized.

[0078] It should be noted that when the image label of the target object is determined by means of the dynamic tree, the construction process of the dynamic tree is a dynamic process. For each target object corresponding to the original image set, each time a branch corresponding to a target object is added to the dynamic tree, the calculation of the attribute weight of the candidate attribute of each target object is re-performed, and the reordering of the candidate attributes of each branch is performed once, until all target objects are added to the dynamic tree.

[0079] S140, determining a target image set according to the target object image corresponding to the target image label.

[0080] The target image label refers to the label required for constructing the target image set, and the target image set can be a training set or a test set.

[0081] Specifically, according to the target object image corresponding to the target image label, the target object containing the target image label in the image label can be determined, the target object is cropped to obtain the target object image, and each target object image is grouped to form the target image set corresponding to the target image label.

[0082] In this embodiment, the dimension of the target image label can be less than the dimension of the image label of each target object determined according to S110-S130. When the target image set corresponding to the target image label is constructed, the target object containing the target image label in the image label can be cropped and added to the target image set. For example, when the target image label is "wearing glasses", the target object with the image label "wearing glasses-white T-shirt-long hair" and the target object with the image label "male-wearing glasses" can be cropped to obtain the target object image and then added to the target object set.

[0083] In this embodiment, if the target object is in the form of a target object detection frame, when the image label of the target object contains the target image label, the original image to which the target object detection frame belongs can be cropped according to the position coordinates of the target object detection frame in the original image to obtain the target object image.

[0084] Further, this training set or test set construction method based on the dynamic tree can intuitively reflect the current missing data scenario, so that the missing scenario can be supplemented in time to obtain a more rich multi-attribute training set or test set.

[0085] The image label determination and the generation of the training set or the test set are performed in the manner of the embodiment, the cost of manual image attribute labeling is reduced, the cumbersome processes such as combined attribute filtering and subsequent image-text matching correction are saved, and the construction efficiency of the training set or the test set is improved. The training set or the test set that is rich in scenes, conforms to reality and is combined with multiple attributes can be constructed, the constructed training set can be used for training of the image-text model, the test set can be used for detecting the image-text matching ability and the image-text understanding ability of the image-text model, and reliable demonstration and data support are provided for training and optimization of the model.

[0086] The technical scheme of the embodiment of the application determines a target object image in an original image, determines a candidate attribute set for the target object image, determines the attribute frequency and the co-occurrence frequency of each candidate attribute in the candidate attribute set, determines the attribute weight of each candidate attribute in the candidate attribute set based on the attribute frequency and the co-occurrence frequency, and selects each candidate attribute in the candidate attribute set according to the attribute weight to obtain a target attribute as the image attribute of the target object image. The technical scheme of the embodiment of the application realizes automatic labeling of the target object attribute in an image, reduces the manual cost of image labeling and classification, and can subsequently realize rapid image attribute classification and construction of a training set and / or a test set with the same image attribute according to the image attribute of each target object in the original image.

[0087] Embodiment two

[0088] Figure 3 A flowchart of a method for determining a target image set is provided for the embodiment two of the application. The embodiment of the application further specifies the process of determining the candidate attribute set matched with the target object, the process of determining the attribute weight of each candidate attribute, and the process of determining the target attribute in each candidate attribute, and adds the process of verifying the image label of the target object.

[0089] As Figure 3 shown, the method comprises:

[0090] S210, target object recognition is performed on an original image to obtain a target object.

[0091] S220, attribute labeling is performed on the target object to obtain an attribute dictionary of the target object.

[0092] S230, word segmentation matching is performed on each attribute in the attribute dictionary to obtain a candidate attribute set composed of at least two candidate attributes.

[0093] S240, for each candidate attribute in the candidate attribute set, the attribute frequency of the candidate attribute is determined according to the number of occurrences of the candidate attribute in each candidate attribute corresponding to each target object and the total number of each candidate attribute corresponding to each target object.

[0094] S250, determining a co-occurrence frequency matching the candidate attribute according to the attribute co-occurrence matrix.

[0095] S260, determining an attribute weight of the candidate attribute according to the attribute frequency of the candidate attribute, the attribute frequency weight, the co-occurrence frequency of the candidate attribute, and the co-occurrence frequency weight.

[0096] S270, determining a target attribute from the candidate attributes as an image label matching the target object according to the attribute weight of each candidate attribute in the candidate attribute set.

[0097] The above embodiment has described the specific process of S210-S270, which will not be repeated here.

[0098] In an optional embodiment, a graph-text model can be used to calculate a graph-text matching degree of the target object and its matching image label. For a target object with a graph-text matching degree less than or equal to a preset graph-text matching degree threshold, the corresponding branch thereof is deleted in the dynamic tree, and the image label thereof is re-determined. For example, the graph-text matching degree threshold can be set to 0.5.

[0099] In another optional embodiment, the embodiment S280-S2110 further provides a multi-model combination verification manner.

[0100] Through the calculation of the graph-text matching degree of the target object and its image label, the strict matching of the target object and its image label can be ensured, thereby improving the accuracy of the image label and the quality of the subsequently constructed training set or test set.

[0101] S280, calculating a graph-text matching degree of the target object and its matching image label by at least two natural language models to obtain at least two initial graph-text matching degrees.

[0102] The natural language model can use different types of graph-text models to respectively understand and calculate the graph-text matching degree, and then perform combination verification. The graph-text model performs deep semantic understanding on the target object and the image label, maps them to the same vector space, and measures the matching degree through vector similarity.

[0103] S290, weighting and summing each initial graph-text matching degree according to the weight of each natural language model to obtain a target graph-text matching degree of the target object.

[0104] In the embodiment, when the image-text matching degrees are calculated by the at least two image-text models, the image-text matching degrees calculated by the image-text models are taken as initial image-text matching degrees, and the initial image-text matching degrees are weighted and summed according to the weights of the image-text models to obtain the final target image-text matching degree of the target object.

[0105] In an optional embodiment, the weights of the image-text models can be fixed values. It can be understood that different image-text models have different understanding abilities for different types of images. For example, the image-text models are a vision-language model (VLM), a multimodal large-language model (MLLM) and a sigmoid loss for language-image pre-training (SigLIP) model respectively. The VLM model and the MLLM model have better understanding ability for image details and are more advantageous for fine-grained image-text matching. The SigLIP has better understanding ability for the overall semantics of an image. Considering the advantages of the image-text models in image understanding, the weights of the image-text models can be set as 3, 1.5 and 0.9 respectively.

[0106] In another optional embodiment, the setting process of the weight of each natural language model can include:

[0107] S1, calculating an image blur normalization value and an image size normalization value of the target object;

[0108] S2, determining the weight of the natural language model according to the image blur normalization value, the image size normalization value and a model bias hyperparameter matched with the natural language model.

[0109] The image blur normalization value can be calculated by the following formula: Wherein, F represents the Sobel gradient variance of the image of the target object, F min = 10 represents that the image is completely blurred, F max = 100 represents that the image is completely clear.

[0110] The image size normalization value can be calculated by the following formula: Wherein, S represents the pixel size of the image of the target object, S min = 20x20, S max = 1024x1024.

[0111] The model bias hyperparameter is a fixed numerical value set in advance, and is set based on the different advantages of different image-text models in image understanding, such as 3, 1.5, and 0.9 in the above embodiment.

[0112] According to the image blur normalization value, the image size normalization value, and the model bias hyperparameter matched with the natural language model, the weight of the natural language model is determined. Specifically, the product of the image blur normalization value, the image size normalization value, and the model bias hyperparameter matched with the natural language model can be taken as the weight of the natural language model.

[0113] In this embodiment, since the image blur normalization value and the image size normalization value of different target objects are usually not the same, the model bias hyperparameter is adjusted based on the specific conditions of the blur and size of all target objects, and is used as the weight of each natural language model for final verification. Thus, the accuracy and precision of model combination verification are improved.

[0114] S2100, determine whether the target image-text matching degree meets the image-text matching degree condition. If yes, execute S2110, otherwise return to execute S220.

[0115] The image-text matching degree condition can mean that the target image-text matching degree is greater than or equal to a preset image-text matching degree threshold.

[0116] Further, when performing word segmentation matching on the multi-dimensional attributes in the attribute dictionary of the target object, multiple candidate attribute sets can be obtained for the same target object, and thus multiple image labels corresponding to the multiple candidate attribute sets can be obtained. At this time, the target image-text matching degree meets the image-text matching degree condition, which means that the image-text matching degree is calculated by using the multi-model combination verification method according to the original candidate attribute set of the target object before word segmentation matching. If the image-text matching degree is greater than or equal to a preset image-text matching degree threshold, it is determined that the image-text matching degree condition is met, otherwise the image-text matching degree condition is not met, and the multiple candidate attribute sets of the target object and the corresponding image labels are all deleted, and the attribute labeling of the target object is performed again.

[0117] S2110, determine a target image set according to the target object image corresponding to the target image label.

[0118] In the embodiment, the target object is obtained by target object recognition on the original image, the attribute dictionary is obtained by attribute tagging on the target object, and the multiple candidate attribute sets are obtained by word segmentation matching on the multi-dimensional attributes in the attribute dictionary. The attribute frequency and co-occurrence frequency of each candidate attribute are calculated respectively, and the attribute weight is determined based on the attribute frequency and co-occurrence frequency, so as to realize the quantitative representation of the scene correlation degree, attribute correlation degree and importance of the candidate attribute, thereby providing a data basis for the selection of the image label of the target object in the subsequent process. For each candidate attribute in the candidate attribute set, the target attribute is selected based on the attribute weight as the image label of the target object, so as to realize the automatic annotation of the image label of the target object, save the cost of manual image annotation, attribute combination and attribute classification, and based on the image label of each target object, the training set or test set is constructed, so as to improve the construction speed and quality of the training set or test set. For the target object and the image label thereof, the graph-text matching degree is calculated through multiple types of graph-text models, the combination verification of multiple models is realized, the strict matching of the target object and the image label thereof is ensured, the accuracy of the image label annotation is improved, so as to improve the data quality of the constructed training set or test set, and provide strong data support for model training and optimization.

[0119] Embodiment three

[0120] Figure 4 A structural schematic diagram of a target image set determination device provided by the third embodiment of the present application is shown in FIG. 3. As shown in FIG. 3, the device comprises: Figure 4

[0121] A candidate attribute set determination module 310 is configured to determine a target object according to an original image, and determine a candidate attribute set matched with the target object;

[0122] An attribute weight determination module 320 is configured to determine the attribute weight of each candidate attribute according to the attribute frequency and co-occurrence frequency of each candidate attribute in the candidate attribute set;

[0123] An image attribute determination module 330 is configured to determine a target attribute from each candidate attribute according to the attribute weight of each candidate attribute in the candidate attribute set, as an image label matched with the target object;

[0124] A target image set determination module 340 is configured to determine a target image set according to the target object image corresponding to the target image label.

[0125] ​The technical scheme of the embodiment of the present application determines a target object in an original image, determines a candidate attribute set for the target object, determines the attribute frequency and co-occurrence frequency of each candidate attribute in the candidate attribute set respectively, determines the attribute weight of each candidate attribute in the candidate attribute set based on the attribute frequency and co-occurrence frequency, filters each candidate attribute in the candidate attribute set according to the attribute weight to obtain a target attribute as the image label of the target object, and finally constructs an image training set or test set based on the image label. The technical scheme of the embodiment of the present application realizes automatic labeling of the attributes of the target object in the image, reduces the labor cost of image labeling and classification, and can realize rapid image attribute classification and construction of a training set and / or test set of the same image attribute according to the image attributes of each target object in the original image.

[0126] On the basis of the above embodiment, the candidate attribute set determination module 310 comprises:

[0127] The target object image determination unit is configured to recognize the target object in the original image to obtain the target object.

[0128] The attribute dictionary determination unit is configured to label the attributes of the target object to obtain an attribute dictionary of the target object.

[0129] The candidate attribute set determination unit is configured to perform word segmentation matching on each attribute in the attribute dictionary to obtain a candidate attribute set composed of at least two candidate attributes.

[0130] The candidate attribute is a single attribute and / or a combined attribute.

[0131] On the basis of the above embodiment, the attribute weight determination module 320 comprises:

[0132] The attribute frequency determination unit is configured to determine the attribute frequency of each candidate attribute in the candidate attribute set according to the number of occurrences of the candidate attribute in each candidate attribute corresponding to each target object and the total number of each candidate attribute corresponding to each target object image.

[0133] The co-occurrence frequency determination unit is configured to determine the co-occurrence frequency matched with the candidate attribute according to the attribute co-occurrence matrix, wherein the elements of the attribute co-occurrence matrix represent the probability of simultaneous occurrence of different candidate attributes.

[0134] The attribute weight determination unit is configured to determine the attribute weight of the candidate attribute according to the attribute frequency, the attribute frequency weight, the co-occurrence frequency of the candidate attribute, and the co-occurrence frequency weight.

[0135] On the basis of the above embodiment, the device further comprises:

[0136] An initial image-text matching degree determination module is configured to calculate image-text matching degrees of the target object and the image label matched with the target object by using at least two natural language models, to obtain at least two initial image-text matching degrees.

[0137] A target image-text matching degree determination module is configured to weight and sum the initial image-text matching degrees according to weights of the natural language models, to obtain a target image-text matching degree of the target object.

[0138] An image-text matching degree judgment module is configured to re-determine the image attribute of the target object if it is determined that the target image-text matching degree does not satisfy the image-text matching degree condition.

[0139] On the basis of the above-mentioned embodiments, optionally,

[0140] A normalization value determination module is configured to calculate an image blur normalization value and an image size normalization value of the target object.

[0141] A weight determination module is configured to determine the weight of the natural language model according to the image blur normalization value, the image size normalization value, and a model bias hyperparameter matched with the natural language model.

[0142] On the basis of the above-mentioned embodiments, optionally, the device further comprises:

[0143] A dynamic tree construction module is configured to construct a dynamic tree according to the candidate attribute set matched with each target object.

[0144] The dynamic tree comprises a root node and branches matched with each target object, and each candidate attribute in the candidate attribute set matched with the target object is a child node in the branch matched with the target object.

[0145] The image attribute determination module 330 comprises:

[0146] A candidate attribute sorting unit is configured to re-sort each candidate attribute in the branch matched with the target object according to an attribute weight of each candidate attribute in the candidate attribute set.

[0147] An image attribute determination unit is configured to take the candidate attributes in the first preset number of layers as the image attribute matched with the target object.

[0148] On the basis of the above-mentioned embodiments, the target image set determination module 340 comprises:

[0149] A target object image determination unit is configured to determine a target object containing a target image label in the image label, to crop the target object to obtain a target object image.

[0150] The target image set determination unit is configured to group each target object image into a target image set corresponding to a target image label.

[0151] The target image set determination apparatus provided by the embodiments of the present application can perform the target image set determination method provided by any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of performing the method.

[0152] Embodiment four

[0153] Figure 5 A structural schematic diagram of an electronic device 10 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application described and / or claimed in this document.

[0154] As shown in Figure 5 The electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11, wherein the memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0155] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, a speaker, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0156] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the determination method of the target image set.

[0157] In some embodiments, the determination method of the target image set can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the determination method of the target image set described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the determination method of the target image set by any other suitable means, such as by means of firmware.

[0158] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0159] Computer programs used to implement the methods of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable processing apparatuses to produce a machine, such that the computer program, when executed by the processor, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, and partially on a machine or entirely on a remote machine or server.

[0160] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0161] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0162] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0163] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0164] It should be understood that the various forms of flow shown above can be reordered, added to, or have steps deleted. For example, the steps described in the present application can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions of the present application can be achieved, and this is not limited herein.

[0165] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for determining a set of target images, characterized in that, include: The target object is determined based on the original image, and a set of candidate attributes matching the target object is determined. The attribute weight of each candidate attribute is determined based on the attribute frequency and co-occurrence frequency of each candidate attribute in the candidate attribute set. Based on the attribute weights of each candidate attribute in the candidate attribute set, the target attribute is determined from among the candidate attributes and used as the image label matching the target object; The target image set is determined based on the target object image corresponding to the target image label.

2. The method according to claim 1, characterized in that, The target object is identified based on the original image, and a set of candidate attributes matching the target object is determined, including: The target object is identified by performing target object recognition on the original image. Tag the attributes of the target object to obtain the attribute dictionary of the target object; Perform word segmentation and matching on each attribute in the attribute dictionary to obtain a candidate attribute set consisting of at least two candidate attributes; The candidate attributes are single attributes and / or combined attributes.

3. The method according to claim 1, characterized in that, Based on the attribute frequency and co-occurrence frequency of each candidate attribute in the candidate attribute set, the attribute weight of each candidate attribute is determined, including: For each candidate attribute in the candidate attribute set, the attribute frequency of the candidate attribute is determined based on the number of times the candidate attribute appears in each candidate attribute corresponding to each target object and the total number of each candidate attribute corresponding to each target object. Based on the attribute co-occurrence matrix, the co-occurrence frequency matching the candidate attribute is determined, wherein the elements of the attribute co-occurrence matrix represent the probability of different candidate attributes occurring simultaneously. The attribute weight of a candidate attribute is determined based on its attribute frequency, attribute frequency weight, co-occurrence frequency, and co-occurrence frequency weight.

4. The method according to claim 1, characterized in that, After determining the target attribute from among the candidate attributes based on the attribute weights of each candidate attribute in the candidate attribute set, and using it as the image label matching the target object, the method further includes: Using at least two natural language models, the image-text matching degree of the target object and the image tags that match the target object is calculated to obtain at least two initial image-text matching degrees. Based on the weights of each natural language model, the initial image-text matching scores are weighted and summed to obtain the target image-text matching score of the target object. If it is determined that the target image-text matching degree does not meet the image-text matching degree condition, then the image attributes of the target object are re-determined.

5. The method according to claim 4, characterized in that, The process of setting the weights for each natural language model includes: Calculate the image blur normalization value and image size normalization value of the target object; The weights of the natural language model are determined based on the image blur normalization value, the image size normalization value, and the model bias hyperparameter matching the natural language model.

6. The method according to any one of claims 1-5, characterized in that, After determining the set of candidate attributes that match the target object, the following steps are also included: A dynamic tree is constructed based on the set of candidate attributes that match each target object; The dynamic tree includes a root node and branches that match each target object. The child nodes in the branches that match the target objects are each candidate attribute in the set of candidate attributes that match the target objects. Based on the attribute weights of each candidate attribute in the candidate attribute set, a target attribute is determined from among the candidate attributes as the image attribute matching the target object, including: For branches that match the target object, the candidate attributes are reordered according to the attribute weights of each candidate attribute in the candidate attribute set. The candidate attributes of the previous preset number of layers are used as image attributes to match the target image.

7. The method according to claim 1, characterized in that, Based on the target object image corresponding to the target image label, determine the target image set, including: Identify the target object whose image label contains the target image label, crop the target object, and obtain the target object image; The images of each target object are combined into a set of target images corresponding to the target image labels.

8. A device for determining a set of target images, characterized in that, include: The candidate attribute set determination module is used to determine the target object based on the original image and to determine the candidate attribute set that matches the target object; The attribute weight determination module is used to determine the attribute weight of each candidate attribute based on the attribute frequency and co-occurrence frequency of each candidate attribute in the candidate attribute set. The image attribute determination module is used to determine the target attribute from among the candidate attributes according to the attribute weight of each candidate attribute in the candidate attribute set, and use it as an image label that matches the target object; The target image set determination module is used to determine the target image set based on the target object image corresponding to the target image label.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method for determining the target image set as described in any one of claims 1-7.

10. A storage medium for storing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the method for determining the target image set as described in any one of claims 1-7.

Images