A data augmentation method, device and medium for digital images

Abstract

The application provides a data enhancement method, device and medium for a digital image, and relates to the technical field of data enhancement of digital images, and the method comprises the following steps: inputting a text description TA input by a user for generating an initial image into a large language model to generate a plurality of initial images; inputting a knowledge graph ZA corresponding to any initial image RA into a preset image classification model W to obtain an image category QA corresponding to the RA; acquiring an image category LA to which the TA belongs in a plurality of image categories corresponding to the W; if the QA and the LA are the same image category, the RA is retained; otherwise, the RA is deleted; the application can realize automatic determination of an initial image category, and filter images according to a target image category associated with a text description, thereby replacing an artificial filtering link, greatly reducing the labor and time cost when generating a large number of images, and improving the application efficiency of data enhancement.

Description

Technical Field

[0001] This invention relates to the field of digital image data enhancement technology, and in particular to a method, apparatus and medium for digital image data enhancement. Background Technology

[0002] In the fields of machine learning and computer vision, model performance is highly dependent on the quality and scale of training data. Sufficient and representative training samples not only help the model learn task-related features more fully but also effectively improve its generalization ability and reduce the risk of overfitting. Among them, image recognition, as one of the core tasks of computer vision, has a particularly large demand for training data. However, in practical applications, collecting image samples by physical means often faces many limitations: on the one hand, the collection process is limited by objective factors such as scene accessibility, equipment cost, and time cycle; on the other hand, even if basic collection is completed, the diversity of samples may be insufficient, failing to cover real application scenarios with different lighting, angles, and noise interference, thus restricting the model's adaptability to complex environments.

[0003] With the development of generative artificial intelligence technology, Large Language Models (LLMs), with their powerful semantic understanding and generation capabilities, are gradually being applied to image generation tasks. By inputting text descriptions (such as images of city traffic signs reflecting rain), LLMs can drive associated image generation modules to directly generate image samples that meet semantic requirements. However, the images generated by current LLMs still suffer from uneven quality: limited by the model's ability to reproduce detailed features and the accuracy of mapping text descriptions to visual features, only a portion of the generated images fully meet actual training requirements. Therefore, to ensure the quality of samples used for data augmentation, images generated by LLMs must be manually screened—especially in large-scale generation scenarios (such as generating tens of thousands or even hundreds of thousands of images at once). Manual screening not only requires significant manpower and time costs but also necessitates the establishment of strict screening criteria to prevent low-quality samples from entering the training process. This situation restricts the efficiency of LLM applications in data augmentation. Summary of the Invention

[0004] To address the aforementioned technical problems, the technical solution adopted by this invention is as follows: According to a first aspect of this application, a data augmentation method for digital images is provided, the method comprising the following steps: S10: Input the text description TA, which is used by the user to generate the initial image, into the large language model to generate several initial images; S20, input the knowledge graph ZA corresponding to any initial image RA into the preset image classification model W to obtain the image category QA corresponding to RA; ZA is obtained based on the element descriptions in RA and the relationships between elements. S30, obtain the image category LA to which TA belongs among several image categories corresponding to W; S40, if QA and LA are in the same image category, keep RA; otherwise, delete RA. W and its corresponding image categories are obtained through the following steps: S01, obtain the overall description text corresponding to each image sample and the element description text corresponding to each element in each image sample; S02, cluster the overall description text corresponding to each image sample to obtain several image categories corresponding to W; S03, take each element in each image sample as a node, and generate the edge between nodes according to the relationship type between any two elements to obtain the knowledge graph corresponding to each image sample; each node is associated with the semantic feature vector corresponding to the text describing the element. S04, use each knowledge graph and its corresponding image category to train the initial image classification model to obtain W.

[0005] According to another aspect of this application, a non-transitory computer-readable storage medium is also provided, wherein at least one instruction or at least one program is stored in the storage medium, and the at least one instruction or at least one program is loaded and executed by a processor to implement the above-described data enhancement method for digital images.

[0006] According to another aspect of this application, an electronic device is also provided, including a processor and the aforementioned non-transitory computer-readable storage medium.

[0007] The present invention has at least the following beneficial effects: The data augmentation method for digital images of this invention automatically determines the category of the initial image by inputting the knowledge graph corresponding to the initial image generated by a large language model into a preset image classification model trained by image sample overall description text clustering and knowledge graph training. It then filters images based on the target image category associated with the text description, replacing the manual screening step. This significantly reduces the manpower and time costs of large-scale image generation and improves the application efficiency of data augmentation. Simultaneously, training the classification model based on the knowledge graph constructed from image elements and the relationships between elements makes the image category determination more closely match the semantic and structural features of the image, ensuring the matching degree between the filtered image and the text description, effectively improving the quality of image samples used for data augmentation, and thus helping to optimize the training effect of the image recognition model. Furthermore, the method of generating initial images based on a large language model breaks through the dependence of traditional data augmentation on original samples, allowing for flexible expansion of the scale and diversity of training samples, compensating for the shortcomings of physically acquired image samples. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 A flowchart illustrating the establishment of a classification model for a data augmentation method for digital images provided in an embodiment of the present invention; Figure 2 This is a flowchart of the initial image selection process for a data augmentation method for digital images provided in an embodiment of the present invention. Detailed Implementation

[0010] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0011] It should be noted that, based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Furthermore, this device and / or practice the method can be implemented using other structures and / or functionalities besides one or more of the aspects set forth herein.

[0012] The following will refer to Figure 1 and Figure 2 The flowchart shown illustrates a data augmentation method for digital images, introducing one such method.

[0013] like Figure 1 As shown, the data augmentation method for digital images may include the following steps: In this embodiment, a classification model can be trained using several prepared images in advance, and the image categories corresponding to the classification model can be established. Specifically, this can include the following steps: S01, obtain the overall description text corresponding to each image sample and the element description text corresponding to each element in each image sample.

[0014] Furthermore, each element in each image sample and its corresponding element description text are obtained through the following steps: S011, input each image sample into the pre-trained YOLOv8 object detection model, and filter out detection boxes with confidence scores higher than the preset confidence threshold; the target corresponding to each detection box is an element in the image sample; and obtain the element category label output by the YOLOv8 object detection model.

[0015] For example, firstly, the confidence threshold of the pre-trained YOLOv8 model is set to a fixed value, such as 0.7 (this threshold is an empirical value and can be adjusted according to needs; it is only used here for explanation and is not intended to impose any restrictions). Input image samples (such as an image showing "a chef stir-frying green vegetables in a black iron wok in front of a kitchen stove"). The model extracts features and detects objects, outputting multiple bounding boxes and their corresponding confidence and category labels—for example, a "chef" bounding box with a confidence of 0.92, a "iron wok" bounding box with a confidence of 0.88, a "vegetables" bounding box with a confidence of 0.95, and a "stove" bounding box with a confidence of 0.85. The "spice bottle" bounding box with a confidence of 0.6 is filtered out because it is below the threshold. Finally, the elements in the image are determined to be "chef," "iron wok," "vegetables," and "stove," and the corresponding category labels are obtained.

[0016] In this step, by setting a confidence threshold, high-reliability detection results are filtered out, effectively eliminating interference targets with low confidence, and ensuring that the identified elements are highly matched with the actual content of the image; at the same time, the obtained element category labels provide basic semantic anchors for subsequent feature analysis and descriptive text generation, avoiding ambiguity in element recognition.

[0017] S012, crop the element region corresponding to each detection box to obtain a local image of the element.

[0018] Based on the coordinates of the detection boxes output by YOLOv8 (e.g., the coordinates of the "iron pot" detection box are (x1=280, y1=320, x2=450, y2=480)), an image cropping tool (such as OpenCV's crop() function) is used to precisely crop the region within the coordinate range of the original image sample, removing background interference and retaining only the local image of the "iron pot". Similarly, the detection boxes of "chef", "vegetables" and "stove" are cropped respectively to obtain the local images of their respective elements.

[0019] By cropping and focusing on the element itself and removing irrelevant background information, subsequent visual feature extraction is only targeted at the main element, avoiding the interference of background noise on the accuracy of features, and laying the foundation for accurately extracting the color, shape, and texture features of the element.

[0020] S013: Input the local image of the element into the pre-trained ResNet50 visual feature extraction model to extract the color features, shape features and texture features of the element.

[0021] The cropped partial image of the "iron pot" is input into a pre-trained ResNet50 model (the top classification layer is removed, but the feature extraction layer is retained). The model extracts multi-dimensional visual features through convolutional and pooling layers: color features are obtained using the mean vector of the RGB color space (e.g., the RGB mean of the iron pot is (56,58,60)) and the HSV color histogram; shape features are obtained by extracting contours through edge detection (Canny algorithm) and calculating Hu moments (e.g., the Hu moment vector of the circular contour of the iron pot), contour area, and aspect ratio; texture features are obtained using the Local Binary Pattern (LBP) histogram (e.g., the LBP feature of the rough metallic texture on the surface of the iron pot), and the contrast and entropy value of the Gray-Level Co-occurrence Matrix (GLCM). Similarly, the light green RGB mean of the "green vegetables," the Hu moments of irregular leaves, and the LBP features of smooth leaf veins are extracted.

[0022] The ResNet50 model can efficiently extract multi-dimensional visual features of elements. Color, shape, and texture features complement each other, comprehensively depicting the visual attributes of elements and avoiding the limitations of single features. This provides a quantitative visual basis for generating accurate and detailed element description text.

[0023] S014. Input the preset Prompt words into the GPT-3.5-turbo model to generate the element description text for the corresponding element. The Prompt words include: element category as {category label}, color feature as {color parameter}, shape feature as {shape parameter}, and texture feature as {texture parameter}.

[0024] Construct a structured prompt by substituting the category labels from S51 and the feature parameters from S53. For example, the prompt for "iron pot" would be: "Element category is {iron pot}, color features are {RGB mean (56,58,60), dark black metallic texture}, shape features are {round bottom, curved handle, Hu moment vector (0.21,0.15,0.08,...)}, texture features are {rough metallic texture, LBP histogram peaks concentrated in low-frequency regions}, please generate a concise and accurate element description text." Input this prompt into the GPT-3.5-turbo model. The model combines semantic understanding and feature information to generate the description text: "Dark black round iron pot with a curved wooden handle, rough metallic texture on the surface, and a wide bottom suitable for stir-frying."

[0025] By combining the category labels of elements with multi-dimensional visual features through Prompt, GPT-3.5-turbo is guided to generate descriptive text that not only fits the category attributes but also accurately reflects visual details, avoiding the generality and ambiguity of the description. The generated high-quality element description text provides a reliable textual foundation for the generation of node semantic features in the subsequent knowledge graph construction, thereby improving the semantic richness of the knowledge graph.

[0026] S02, cluster the overall description text corresponding to each image sample to obtain several image categories corresponding to W.

[0027] Furthermore, step S02 includes the following steps: S021, the jieba word segmentation tool is used to segment the overall description text corresponding to each image sample.

[0028] First, collect the overall descriptive text of the image samples. For example, the overall description of an image sample is "A barista is making a latte at the bar counter in a coffee shop using a coffee machine, and ceramic coffee cups are placed on the bar counter." Use the jieba word segmentation tool to segment the text. First, load the Chinese word segmentation dictionary and perform precise mode word segmentation to obtain the initial segmentation result: "Barista / at / coffee shop / bar counter / using / coffee machine / making / latte coffee / , / bar counter / on / placed / ceramic coffee cups". Then, filter out stop words such as "at", "using", and "placed" as well as punctuation marks such as commas to finally obtain the processed word segmentation sequence: "Barista / coffee shop / bar counter / coffee machine / making / latte coffee / bar counter / placed / ceramic coffee cups".

[0029] Jieba segmentation breaks down continuous text into semantic units, filtering out meaningless words and punctuation. This simplifies the text structure while preserving core semantic information, avoiding redundant content from interfering with subsequent semantic feature extraction, and laying the text foundation for accurate generation of semantic feature vectors.

[0030] S022, input the processed overall descriptive text into the pre-trained BERT-base-uncased model to obtain the semantic feature vector of the overall descriptive text.

[0031] First, the processed word segmentation sequence is processed for the input format of the BERT model by adding a [CLS] tag at the beginning of the sequence and a [SEP] tag at the end, and then tokenization is performed (the word segmentation is converted into a token that the model can recognize). The processed input sequence is then fed into the pre-trained BERT-base-uncased model, which performs semantic encoding through a multi-layer Transformer encoder. Finally, the output vector corresponding to the [CLS] tag is extracted as the semantic feature vector of the overall descriptive text (768-dimensional).

[0032] For example, after the text of the above coffee shop scene is processed by the model, a 768-dimensional vector containing core semantics such as "barista + coffee shop + coffee machine" is obtained, which can accurately represent the semantic connotation of the text.

[0033] The BERT model can capture the contextual semantic relationships of text. The semantic feature vectors generated are more semantically representative than the traditional bag-of-words model. They can accurately reflect the core of the overall text description and provide highly recognizable feature basis for subsequent clustering, avoiding clustering bias due to insufficient semantic representation.

[0034] S023, the K-means++ clustering algorithm is used to cluster all semantic feature vectors to obtain several image categories corresponding to W.

[0035] First, the semantic feature vectors of all overall descriptive texts are preprocessed (e.g., normalized). Then, the K-means++ algorithm is used to initialize the cluster centers (selecting the initial centers that are farther away by probability sampling to avoid the local optimum problem of random initialization in K-means). Using Euclidean distance as the vector similarity metric, the curve of the sum of squared clustering errors (SSE) changing with the number of clusters K is analyzed by the elbow rule to determine the optimal value of K (e.g., K=4). Then, the process of "assigning samples to the nearest cluster center → updating cluster centers" is iteratively executed until the cluster centers are stable.

[0036] For example, the semantic vectors of texts such as "barista making latte" and "customer enjoying Americano in coffee shop" are clustered into "coffee shop beverage making scene", and "teacher lecturing in classroom with whiteboard" and "student taking notes in classroom" are clustered into "classroom teaching scene", ultimately resulting in several image categories corresponding to W.

[0037] K-means++'s optimized initialization strategy improves the stability and accuracy of clustering. Combined with the high discriminativeness of semantic feature vectors, the clustering results can accurately match the contextual semantics of the text, avoiding the subjectivity of manually defined categories. This constructs a logically consistent image category system that fits the actual scene, providing reliable label support for subsequent classification model training.

[0038] S03, treat each element in each image sample as a node, and generate edges between nodes according to the relationship type between any two elements to obtain the knowledge graph corresponding to each image sample; each node is associated with the semantic feature vector corresponding to the text describing the element.

[0039] Furthermore, step S03 includes the following steps: S031, obtain the core information of each element in each image sample; the core information includes: the element category label, detection box coordinates and confidence score output by the YOLOv8 object detection model, and the corresponding element description text.

[0040] In this embodiment, taking the image sample "a little girl watering pink roses with a watering can in a garden" as an example, the core information of each element is extracted from the YOLOv8 detection results: the category label of "little girl" is "person", the detection box coordinates are (x1=120, y1=80, x2=280, y2=450), the confidence score is 0.96, and the element description text is "a little girl with pigtails and wearing a pink dress"; the category label of "watering can" is "tool", and the detection box coordinates are (x1=260, y1=220, x2=320, y2=300). The confidence score is 0.92, and the element description text is "blue plastic watering can, spout facing upwards"; the category label for "rose" is "plant", the bounding box coordinates are (x1=350, y1=180, x2=480, y2=350), the confidence score is 0.95, and the element description text is "blooming pink rose, petals layered upon layer"; the category label for "garden" is "scene", the bounding box coordinates are (x1=0, y1=0, x2=600, y2=600), the confidence score is 0.88, and the element description text is "an outdoor garden full of green plants and flowers".

[0041] This step integrates the semantic tags, spatial locations, and textual descriptions of elements, providing comprehensive and accurate basic data for subsequent calculation of geometric parameters and determination of element relationships, thus avoiding deviations in relationship determination due to missing information.

[0042] S032, calculate the center distance d, relative azimuth angle θ, and overlap area ratio s of any two elements A and B based on the coordinates of the detection box.

[0043] In this embodiment, the center coordinates of each element detection box are first calculated (center x = (x1 + x2) / 2, center y = (y1 + y2) / 2), for example: the center coordinates of "little girl" are (200, 265), the center coordinates of "watering can" are (290, 260), and the center coordinates of "rose" are (415, 265); the center distance d is calculated using the Euclidean distance formula, such as d = 90.14 pixels for the little girl and the watering can; the relative azimuth angle θ is calculated with the center of element A as the origin, and the azimuth angle of the center of element B is calculated (range 0°-360°), such as θ≈357° (almost to the right) for the watering can and θ≈0° (to the right) for the rose; the overlap area ratio s is calculated as the ratio of the intersection area to the union area of ​​the two detection boxes, and if the two boxes have no intersection, then s = 0 (e.g., the detection boxes of the little girl and the rose have no overlap, so s = 0).

[0044] By quantifying the spatial geometric relationships between elements, we can provide an objective spatial basis for relationship determination, making up for the limitations of relying solely on semantic descriptions and allowing the determination of element relationships to better align with the visual spatial characteristics of images.

[0045] S033, input the preset relationship determination prompt into the pre-trained GPT-4 model, and perform cross-validation with the relationship candidate set output by the pre-trained VisualRelationship Detection model to determine the final relationship type; the relationship determination prompt includes: given that the category of element A is {the category of A}, the description is {the element description text of A}, the category of element B is {the category of B}, the description is {the element text description of B}, the center distance between A and B in the image sample is {d}, the relative azimuth angle is {θ}, and the overlap area ratio is {s}, please determine the relationship type between A and B: only A has a one-way action association with B, A and B have a two-way association, and there is no association.

[0046] For "little girl" (A) and "watering can" (B), construct a prompt: "Given that element A is categorized as {person} and described as {a little girl with pigtails and wearing a pink dress}, element B is categorized as {tool} and described as {a blue plastic watering can with the spout pointing upwards}, the center distance between A and B in the image sample is {90.14 pixels}, the relative azimuth angle is {357°}, and the overlap area is {0}, please determine the relationship type between A and B: only A has a one-way action association with B, A and B have a two-way association, or there is no association."

[0047] The Prompt was input into the GPT-4 model, yielding a preliminary determination of "only A has a one-way action association with B (the little girl is holding a watering can)". Simultaneously, the image was input into the VRD model, outputting candidate relation sets of "little girl - hold - watering can (confidence 0.93)" and "little girl - near - watering can (confidence 0.98)". Cross-validation confirmed the final relation type as "only A has a one-way action association with B", with the semantic meaning of "holding". For "watering can" and "rose", the GPT-4 model determined "only the watering can has a one-way action association with the rose (the watering can waters the rose)", and the VRD model output "watering can - water - rose (confidence 0.91)". Cross-validation confirmed the relation type and semantic meaning.

[0048] In this step, the semantic understanding capability of GPT-4 is combined with the visual relationship detection capability of the VRD model for cross-validation. This not only avoids the problem of pure text semantic judgment being divorced from image vision, but also makes up for the lack of semantic understanding of pure visual models, thus greatly improving the accuracy and reliability of element relationship judgment.

[0049] S034. If A and B are action-related, then a directed edge with an arrow is established between the two corresponding nodes of A and B, with the arrow pointing from the active element node to the passive element node, and the association semantics are marked on the edge; if it is a bidirectional relationship, then a bidirectional directed edge is established and the association semantics are marked; if there is no association, then no edge is established between the nodes.

[0050] For the one-way relationship between "little girl" and "watering can", a directed edge with an arrow is created between the "little girl" node and the "watering can" node (little girl → watering can), labeled "holding". For the one-way relationship between "watering can" and "rose", a directed edge is created (watering can → rose), labeled "watering". For the one-way relationship between "little girl" and "garden", it is determined to be "little girl is in garden", so a directed edge is created (little girl → garden), labeled "is in". Since there is no direct action relationship between "little girl" and "rose", no edge is created. This ultimately forms a knowledge graph containing nodes, directed edges, and semantic annotations.

[0051] In this step, the edges of the knowledge graph are constructed in a structured manner according to the relationship type, clearly presenting the semantic association direction and specific relationship between elements. This enables the knowledge graph to accurately map the semantic logical structure of the image, providing structured support for the subsequent classification model to learn the deep semantics of the image and improving the model's ability to understand the image scene.

[0052] S04, use each knowledge graph and its corresponding image category to train the initial image classification model to obtain W.

[0053] In this embodiment, a graph convolutional network (GCN) is selected as the initial image classification model (or a graph attention network (GAT), which can strengthen the attention weights between nodes). This model can effectively process the graph structure data of knowledge graphs, and aggregate the feature information of nodes and adjacent nodes through multiple graph convolutional layers to output the probability distribution of image categories.

[0054] The knowledge graph of each image sample is transformed into an input format that the model can recognize—the node feature matrix is ​​filled with semantic feature vectors of the element description text (such as 768-dimensional BERT encoded vectors), and the adjacency matrix is ​​constructed according to the edge relationships of the knowledge graph (the position with an edge is 1, the position without an edge is 0, and the weighted edge can be filled with the similarity value of the associated semantics); at the same time, the image categories obtained by clustering (such as "garden gardening scene" and "coffee shop beverage making scene") are one-hot encoded as the model's label data.

[0055] The learning rate was set to 0.001 (using the AdamW optimizer), the batch size to 32, and the number of iterations to 100. The cross-entropy loss function was used to calculate the error between the predicted class and the true class, and an early stopping strategy (training was stopped if the validation set loss did not decrease for 5 consecutive rounds) was used to avoid overfitting.

[0056] The preprocessed knowledge graph data is input into the GCN model. Node features are aggregated through graph convolutional layers (e.g., the "little girl" node aggregates features from neighboring nodes such as "watering can" and "rose"), and then mapped to class probabilities through fully connected layers. Model parameters are optimized during backpropagation to gradually reduce the loss value. For example, given the knowledge graph "a little girl is watering roses in the garden with a watering can," the model initially predicts the class as "outdoor leisure scene" (probability 0.6), which is gradually optimized to "garden horticulture scene" (probability 0.98) after training.

[0057] Using 20% ​​of the image samples as the validation set, the model performance was evaluated by accuracy and F1 score. If overfitting occurred, a Dropout layer was added (dropout rate set to 0.3). Finally, a pre-defined image classification model W with convergence and strong generalization ability was obtained.

[0058] By using the structured semantics of knowledge graphs as model input, the classification model not only learns the features of individual elements but also captures the logical relationships between elements. Compared to traditional classification models based on pixel or single text features, it has stronger semantic understanding capabilities and significantly improves the accuracy of image category determination. At the same time, the trained model W can provide a reliable basis for category determination for subsequent screening of images generated by large language models, ensuring that the selected images are highly matched with the text descriptions, effectively improving the quality of data-enhanced samples, and thus optimizing the training effect of downstream image recognition models.

[0059] The steps S01-S04 above only need to be performed once in this scheme. After obtaining W and the corresponding image categories, the following steps can be performed: Figure 2 The steps shown are as follows: S10: Input the text description TA, which is used by the user to generate the initial image, into the large language model to generate several initial images.

[0060] In this embodiment, the text description TA input by the user must contain clear scene, element and relationship information, for example: TA is "an adult is holding a golden retriever on the beach". Select a large language model with integrated image generation capabilities (such as GPT-4 combined with DALL-E 3, or Wenxin Yiyan combined with Wenxin Yige), input TA as the generation instruction into the model, set the number of images to be generated (such as 500 images), resolution (such as 1024×1024) and other parameters, and the model generates several initial images based on the text semantics (for example: generating images with different compositions and lighting containing the elements of "human + golden retriever + beach + sea").

[0061] This step breaks through the limitations of traditional data augmentation that relies on original image samples. It can quickly generate large-scale and diverse initial images without physical collection, which not only greatly reduces the time and cost of sample collection, but also covers scenes with different compositions and environmental details, making up for the problem of insufficient diversity of physical samples and providing a sufficient sample foundation for subsequent data augmentation.

[0062] S20, input the knowledge graph ZA corresponding to any initial image RA into the preset image classification model W to obtain the image category QA corresponding to RA; ZA is obtained based on the element descriptions in RA and the relationships between elements.

[0063] Based on the preset image classification model W obtained in steps S50-S80 above, ZA is transformed into an input format recognizable by model W (GCN architecture)—a node feature matrix filled with semantic vector elements, and an adjacency matrix constructed based on edge relationships. After input, model W aggregates node and adjacent node features through graph convolutional layers (e.g., the "adult" node aggregates the association features of "golden retriever" and "beach"), outputs the category probability distribution through fully connected layers, and finally determines the image category QA corresponding to RA (e.g., "people-pet beach interaction scene"). ZA can be obtained using the method in step S70, which will not be elaborated here.

[0064] By using the knowledge graph ZA, the visual content of RA is transformed into structured semantic data, avoiding the limitations of models that rely solely on pixel features to determine categories. This makes the QA output more closely aligned with the deep semantics of the image and the logic of the scene. At the same time, relying on the pre-trained model W, automated category determination is achieved, replacing the initial manual screening process. This significantly improves the efficiency of determination, and the consistency and accuracy of the determination results are higher, providing a reliable basis for subsequent image screening and effectively ensuring the quality of data-enhanced samples.

[0065] S30, obtain the image category LA to which TA belongs among several image categories corresponding to W.

[0066] Furthermore, step S30 includes the following steps: S31, obtain the central semantic feature vector corresponding to each image category of W, so as to obtain the list of central semantic feature vectors C = (C1, C2, ..., C...). i C n ), i=1,2,…,n; C i Let C be the central semantic feature vector of the i-th image category corresponding to W, and n be the number of image categories corresponding to W; i It is obtained from the overall descriptive text included in the i-th image category corresponding to W.

[0067] First, extract the overall descriptive text of all image samples under each image category corresponding to W—for example, the first image category of W, "People-Pets Beach Interaction Scene," contains 300 overall descriptive texts (such as "An adult is holding a golden retriever on the beach" and "A child is walking a teddy dog ​​at the beach"). Perform the same semantic encoding process as S62 on each text: after jieba segmentation and stop word filtering, input it into the pre-trained BERT-base-uncased model to extract a 768-dimensional semantic feature vector. Calculate the mean vector of all semantic feature vectors under this category, which is the central semantic feature vector C of this category. i (For example, C1 for "Character-Pet Beach Interaction Scene" is the one-dimensional mean of 300 text vectors); similarly, calculate C1 for all categories such as "Pure Beach Scene" and "Character-Pet Indoor Interaction Scene". i This ultimately forms the central semantic feature vector list C.

[0068] Construct the central semantic feature vector C using the mean of all text semantic vectors under the category. i It can accurately represent the core semantic connotation of each image category, avoid semantic bias caused by single text, and provide a representative reference benchmark for subsequent similarity matching between TA and category, ensuring the rationality of the matching logic.

[0069] S32, obtain the maximum similarity η between the semantic feature vector corresponding to TA and each central semantic feature vector in C. max .

[0070] First, semantic encoding is performed on the user-input text description TA (e.g., "an adult is holding a golden retriever on the beach"). After jieba segmentation and stop word filtering, the text is input into the BERT-base-uncased model to generate a semantic representation of C. i A 768-dimensional semantic feature vector VT with consistent dimensions; the cosine similarity algorithm is used to calculate the similarity between VT and each C in list C. i The similarity (cosine similarity ranges from -1 to 1, with values ​​closer to 1 indicating greater semantic similarity) is calculated. For example, VT has a similarity of 0.85 with C1 (a person-pet seaside interaction scene), 0.32 with C2 (a pure seaside scenery scene), and 0.48 with C3 (a person-pet indoor interaction scene). The maximum similarity value, η, is selected from all similarity results. max =0.85.

[0071] By quantifying the semantic association between TA and each image category using cosine similarity, it is more objective and accurate than subjective human judgment. Furthermore, relying on a unified semantic encoding method ensures the consistency between vector dimension and semantic space, enabling the similarity calculation results to truly reflect the matching degree between TA and category, and providing a quantitative basis for determining LA.

[0072] S33, if η max If the similarity is greater than the first preset threshold, then η will be... max The corresponding image category is determined to be LA.

[0073] In this embodiment, the first preset similarity threshold is set according to the actual application scenario (e.g., set to 0.7, which can be adjusted through validation set optimization; it should be noted that 0.7 is only an example, and this threshold is an empirical value); compared with η max With threshold: if η max =0.85>0.7, then η max The corresponding "character-pet beachside interaction scene" is determined to be LA; if a certain TA's η max =0.65 < 0.7, then it is determined that there is no matching category (which may trigger subsequent logic such as re-entering TA or expanding image category).

[0074] By establishing clear matching and screening criteria through preset similarity thresholds, categories with low semantic relevance can be effectively filtered out, avoiding misclassification of categories that do not conform to the core needs of TA as LA. At the same time, it ensures that LA is the image category that best matches the semantics of TA, providing an accurate and unified target benchmark for subsequent RA screening, ensuring that the screened images are highly matched with user needs, and improving the effectiveness of data augmentation samples.

[0075] S40: If QA and LA are in the same image category, retain RA; otherwise, delete RA.

[0076] First, clarify the category determination criteria for QA and LA. QA is the category identifier (such as category ID or category name) of the initial image RA output by the preset image classification model W, and LA is the category identifier of the target image associated with the text description TA. The two adopt a unified category coding rule (for example, category ID: ID=001 for "People-Pet Seaside Interaction Scene" and ID=002 for "Pure Seaside Scene").

[0077] Taking batch processing as an example: Assume the large language model generates 10,000 initial images. After processing by S20, each initial image corresponds to a unique QA (e.g., QA=001 for 3,800 initial images, QA=002 for 2,200 initial images, and other QA categories). S30 determines LA=001 ("People-Pet Beach Interaction Scene"). The category identifier of each initial image's QA is compared with that of LA: if an initial image's QA=001 matches LA, then that RA is stored in the valid sample library; if an initial image's QA=002 ("Pure Beach Scene") or other non-001 categories, then that RA is directly deleted from the sample set. Finally, 3,800 valid samples meeting the requirements are selected from the 10,000 initial images for subsequent data augmentation.

[0078] Automated initial image selection through direct comparison of category identifiers can replace manual selection in most traditional large-scale generation scenarios, significantly reducing manpower and time costs and greatly improving the efficiency of data augmentation processes. Simultaneously, using the consistency of QA and LA categories as the core selection criterion ensures that the selected RAs (Image Augmentation References) highly match the core needs of the user's input text description TA, effectively filtering out low-quality images with semantic biases and missing elements, thus guaranteeing the quality and purity of the data augmentation sample library. Furthermore, unified category determination rules avoid the subjectivity and inconsistencies of manual selection, giving the selection results good stability and reliability. This provides high-quality, highly relevant training samples for downstream image recognition models, thereby contributing to the improvement of model generalization ability and recognition accuracy.

[0079] In an exemplary embodiment, if the descriptive text entered by the user is inaccurate, there may be a large number of unqualified initial images generated. Therefore, after step S40, the method may further include the following steps: S50, if NUM1 / NUM>σ, then randomly select m image samples from the image samples corresponding to LA to obtain a random image sample list D=(D1, D2, …, D…). j Dm ), j=1,2,…,m; D j is the j-th image sample obtained randomly; NUM1 is the number of initial images to be deleted, NUM is the total number of initial images; σ is the preset deletion ratio threshold.

[0080] First, calculate the total number of initial images NUM (e.g., 1000 initial images generated by a large language model) and the number of deleted initial images NUM1 (e.g., 700 images deleted due to QA and LA mismatch), and calculate the deletion ratio NUM1 / NUM = 0.7. The preset deletion ratio threshold σ is set according to actual needs (e.g., σ = 0.5, which means that if the deletion ratio exceeds 50%, it is determined that the user description text may be inaccurate; it should be noted that 0.5 is only an example and is not a restriction on σ to be 0.5). Since 0.7 > 0.5, the following operation is triggered: randomly select m image samples (m is set to 20) from the sample library of the image category corresponding to LA (e.g., "people-pet beach interaction scene") to obtain list D, for example: D contains image samples such as "adults holding golden retrievers on the beach" and "children walking teddy dogs on the beach".

[0081] The accuracy of user description text is judged by deleting a threshold ratio. When the qualified rate of generated images is too low, reference samples are automatically selected from the target category sample library to provide a basis for subsequent optimization of prompt text. This avoids the continuous failure of generated images due to ambiguous user descriptions and improves the fault tolerance and efficiency of data augmentation.

[0082] S51, randomly select several sets of elements with relationships from each element corresponding to each image sample in D to obtain a list set of element groups E = (E1, E2, ..., E...). j , ..., E m ); E j D j The corresponding list of elements; E j =(E j，1 E j，2 , ..., E j，p , ..., E j，q ), p=1,2,…,q; E j，p For from D j For each element, the p-th group of elements with a relationship is randomly selected, and q is the number of groups of elements with a relationship.

[0083] With D j Taking "an adult cuddling a golden retriever on the beach" as an example, the knowledge graph contains related element groups including (adult, golden retriever, cuddling), (adult, beach, standing), and (golden retriever, beach, lying down); randomly selecting q=2 element groups, i.e., E j= ((adult, golden retriever, holding), (adult, beach, standing)); Perform the same operation on 20 image samples in D to obtain a list set of element groups E, which contains the relational element groups of each sample.

[0084] Extracting real and effective relational element groups from reference samples and capturing the core element association features of target category images provides specific semantic material for generating accurate prompt text, thus avoiding generalized and ambiguous prompt text.

[0085] S52, based on the category labels, relationships, and text descriptions of elements within each element group in E, perform deduplication on the elements in E to obtain a deduplicated list of element groups F = (F1, F2, ..., F...). u F v ), u=1,2,…,v;F u Let v be the u-th element group obtained after deduplication, and v be the number of element groups obtained after deduplication.

[0086] Furthermore, step S52 includes the following steps: S521, for any two element groups EA and EB in E, if the two element category labels in EA are the same as the two element category labels in EB, the corresponding relationship is the same, and the element text description similarity of the elements with the same element category labels is greater than the second preset similarity threshold, then delete EA or EB.

[0087] Let the second preset similarity threshold be an empirical value, which can be set to 0.8 (using cosine similarity to calculate the semantic similarity of the element text descriptions); take EA = (adult, golden retriever, holding) (element description: "adult: man in short sleeves" "golden retriever: golden long-haired dog"), EB = (adult male, golden retriever, holding) (element description: "adult male: adult in casual wear" "golden retriever: golden pet dog"); compare the element category labels (both "person" and "pet dog") and the relationship (both "holding"), and the text similarity between "adult" and "adult male" is 0.92 > 0.8, and the similarity between "golden retriever" is 0.95 > 0.8, so delete EB; if EA = (child, teddy dog, leashed) and EB = (adult, golden retriever, holding), because the element category labels or relationships are different, then keep both groups.

[0088] By implementing clear quantitative rules, element groups can be deduplicated, avoiding the subjectivity of manual screening, ensuring the consistency and accuracy of deduplication results, and preserving the diverse element association patterns in the target category.

[0089] S53, based on the category label, relational semantics, and element text description of each element group in F, generate the prompt description text for each element group, thus obtaining the prompt description text list G = (G1, G2, ..., G...). u , ..., G v ); G u For F u The corresponding prompt description text.

[0090] With F u = (Adult, Golden Retriever, Holding) For example, combining the element text description, generate the prompt description text: "An adult in short sleeves holding a long-haired golden retriever"; using F u =(child, teddy dog, on a leash) generates: "A child with a ponytail is on a leash for a brown curly-haired teddy dog"; the final result is a list G containing the precise hint text for each element group.

[0091] By converting structured element group information into natural language prompt text, specific details are provided to optimize the user input TA, solve the problem of vague TA description, and improve the accuracy of subsequent image generation.

[0092] S54: After concatenating G and TA, input them into the large language model to generate several initial images; proceed to S20.

[0093] The user's original input TA was "hugging a dog at the beach". The prompt text in G is concatenated with TA to obtain the new input text: "hugging a dog at the beach, an adult in short sleeves is holding a golden retriever with long hair, and a child with a ponytail is leading a brown curly-haired teddy bear". The concatenated text is input into a large language model (such as DALL-E 3) to generate 1000 new initial images. Then, S20 and subsequent filtering steps are repeated.

[0094] By supplementing precise and detailed prompts to optimize user input, the large language model is guided to generate images that better fit the target category, reducing the proportion of unqualified images. At the same time, a closed-loop optimization process is formed, which can automatically correct generation biases without manual intervention, further improving the automation level of data augmentation and sample quality.

[0095] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0096] Embodiments of the present invention also provide a non-transitory computer-readable storage medium that can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a method in the method embodiments, wherein the at least one instruction or the at least one program is loaded and executed by the processor to implement the method provided in the above embodiments.

[0097] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0098] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0099] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0100] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0101] Embodiments of the present invention also provide an electronic device, including a processor and the aforementioned non-transitory computer-readable storage medium.

[0102] The electronic device is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments in this application.

[0103] Electronic devices are manifested in the form of general-purpose computing devices. Components of an electronic device may include, but are not limited to: at least one processor, at least one memory, and a bus connecting different system components (including memory and processor).

[0104] The memory stores program code that can be executed by the processor, causing the processor to perform the steps in the various embodiments described in this specification.

[0105] The memory may include readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory, and may further include read-only memory (ROM).

[0106] The memory may also include programs / utilities having a set (at least one) of program modules, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0107] A bus can represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus that uses any of the various bus structures.

[0108] Electronic devices can also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable user interaction with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be achieved through input / output (I / O) interfaces. Furthermore, electronic devices can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapters. The network adapter communicates with other modules of the electronic device via a bus. It should be understood that other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0109] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0110] Embodiments of the present invention also provide a computer program product including program code, which, when the program product is run on an electronic device, causes the electronic device to perform the steps of the methods described above in various exemplary embodiments of the present invention.

[0111] While specific embodiments of the invention have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and are not intended to limit the scope of the invention. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the invention.

Claims

1. A data augmentation method for digital images, characterized in that, The method includes the following steps: S10: Input the text description TA, which is used by the user to generate the initial image, into the large language model to generate several initial images; S20, input the knowledge graph ZA corresponding to any initial image RA into the preset image classification model W to obtain the image category QA corresponding to RA; ZA is obtained based on the element descriptions in RA and the relationships between elements. S30, obtain the image category LA to which TA belongs among several image categories corresponding to W; S40, if QA and LA are in the same image category, keep RA; otherwise, delete RA. W and its corresponding image categories are obtained through the following steps: S01, obtain the overall description text corresponding to each image sample and the element description text corresponding to each element in each image sample; S02, cluster the overall description text corresponding to each image sample to obtain several image categories corresponding to W; S03, take each element in each image sample as a node, and generate the edge between nodes according to the relationship type between any two elements to obtain the knowledge graph corresponding to each image sample; each node is associated with the semantic feature vector corresponding to the text describing the element. S04, use each knowledge graph and its corresponding image category to train the initial image classification model to obtain W.

2. The data enhancement method for digital images according to claim 1, characterized in that, Step S02 includes the following steps: S021, the jieba word segmentation tool is used to segment the overall description text corresponding to each image sample; S022, input the processed overall description text into the pre-trained BERT-base-uncased model to obtain the semantic feature vector of the overall description text; S023, the K-means++ clustering algorithm is used to cluster all semantic feature vectors to obtain several image categories corresponding to W.

3. The data enhancement method for digital images according to claim 1, characterized in that, Each element in each image sample and its corresponding element description text are obtained through the following steps: S011, each image sample is input into the pre-trained YOLOv8 object detection model, and detection boxes with confidence scores higher than the preset confidence threshold are selected; the target corresponding to each detection box is an element in the image sample; And obtain the element category labels output by the YOLOv8 object detection model; S012, crop the element region corresponding to each detection box to obtain a local image of the element; S013, input the local image of the element into the pre-trained ResNet50 visual feature extraction model to extract the color features, shape features and texture features of the element; S014, Input the preset Prompt words into the GPT-3.5-turbo model to generate the element description text for the corresponding element; Prompt prompts include: element category as {category label}, color feature as {color parameter}, shape feature as {shape parameter}, and texture feature as {texture parameter}.

4. The data enhancement method for digital images according to claim 3, characterized in that, Step S03 includes the following steps: S031, Obtain the core information of each element in each image sample; the core information includes: the element category label, detection box coordinates and confidence score output by the YOLOv8 object detection model, and the corresponding element description text; S032, calculate the center distance d, relative azimuth angle θ, and overlap area ratio s of any two elements A and B based on the coordinates of the detection box; S033, input the preset relationship determination prompt into the pre-trained GPT-4 model, and perform cross-validation with the relationship candidate set output by the pre-trained VisualRelationship Detection model to determine the final relationship type; the relationship determination prompt includes: given that the category of element A is {the category of A}, the description is {the element description text of A}, the category of element B is {the category of B}, the description is {the element text description of B}, the center distance between A and B in the image sample is {d}, the relative azimuth angle is {θ}, and the overlap area ratio is {s}, please determine the relationship type between A and B: only A has a one-way action association with B, A and B have a two-way association, and there is no association; S034. If A and B are in a one-way action relationship, then a directed edge with an arrow is established between the two corresponding nodes of A and B, with the arrow pointing from the active element node to the passive element node, and the association semantics are marked on the edge; if it is a two-way relationship, then a two-way directed edge is established and the association semantics are marked; if there is no relationship, then no edge is established between the nodes.

5. The data enhancement method for digital images according to claim 1, characterized in that, Step S30 includes the following steps: S31, obtain the central semantic feature vector corresponding to each image category of W, so as to obtain the list of central semantic feature vectors C = (C1, C2, ..., C...). i C n ), i=1,2,…,n; C i Let C be the central semantic feature vector of the i-th image category corresponding to W, and n be the number of image categories corresponding to W; i It is obtained based on the overall descriptive text included in the i-th image category corresponding to W; S32, obtain the maximum similarity η between the semantic feature vector corresponding to TA and each central semantic feature vector in C. max ; S33, if η max If the similarity is greater than the first preset threshold, then η will be... max The corresponding image category is determined to be LA.

6. The data enhancement method for digital images according to claim 1, characterized in that, After step S40, the method further includes the following steps: S50, if NUM1 / NUM>σ, then randomly select m image samples from the image samples corresponding to LA to obtain a random image sample list D=(D1, D2, …, D…). j D m ), j=1,2,…,m; D j The j-th image sample is randomly obtained; NUM1 is the number of initial images to be deleted, NUM is the total number of initial images; σ is the preset deletion ratio threshold; S51, randomly select several sets of elements with relationships from each element corresponding to each image sample in D to obtain a list set of element groups E = (E1, E2, ..., E...). j , ..., E m ); E j D j The corresponding list of elements; E j =(E j，1 E j，2 , ..., E j，p , ..., E j，q ), p=1,2,…,q; E j，p For from D j For each element, the p-th group of elements with a relationship is randomly selected, and q is the number of groups of elements with a relationship. S52, based on the category labels, relationships, and text descriptions of elements within each element group in E, perform deduplication on the elements in E to obtain a deduplicated list of element groups F = (F1, F2, ..., F...). u F v ), u=1,2,…,v;F u Let u be the u-th element group obtained after deduplication, and v be the number of element groups obtained after deduplication. S53, based on the category label, relational semantics, and element text description of each element group in F, generate the prompt description text for each element group, thus obtaining the prompt description text list G = (G1, G2, ..., G...). u , ..., G v ); G u For F u The corresponding prompt description text; S54: After concatenating G and TA, input them into the large language model to generate several initial images; proceed to S20.

7. The data enhancement method for digital images according to claim 6, characterized in that, Step S52 includes the following steps: S521, for any two element groups EA and EB in E, if the two element category labels in EA are the same as the two element category labels in EB, the corresponding relationship is the same, and the element text description similarity of the elements with the same element category labels is greater than the second preset similarity threshold, then delete EA or EB.

8. A non-transitory computer-readable storage medium, wherein the storage medium stores at least one instruction or at least one program segment, characterized in that, The at least one instruction or the at least one program segment is loaded and executed by the processor to implement the data augmentation method for digital images as described in any one of claims 1-7.

9. An electronic device, characterized in that, Includes a processor and the non-transitory computer-readable storage medium as described in claim 8.

Images