Training data set processing method and device, electronic equipment and storage medium

Abstract

The invention discloses a training data set processing method and device, electronic equipment and a storage medium, and belongs to the technical field of data processing. The method comprises the following steps: performing element identification on each target image in a target image data set to obtain distribution results of different element categories; determining a target element category of which the proportion is lower than a target threshold according to the distribution result, and determining a target image containing the target element category as a sparse sample image; generating a target text description according to the features of the environment and the character object in the sparse sample image, and inputting the target text description into an image generation model to obtain a plurality of augmented images corresponding to the sparse sample image; and constructing a training data set according to the target image data set and the plurality of augmented images. The number of sparse samples can be supplemented in a targeted mode, the number and diversity of minority class samples are enriched, all kinds of samples in the training data set are more balanced, and the accuracy of model detection is improved.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to a training dataset processing method, apparatus, electronic device and storage medium. Background Technology

[0002] With the acceleration of digital transformation in the financial industry, object detection technology is being used more and more in financial scenarios. In places such as bank branches and financial service halls, object detection can accurately identify the identity of people in images, distinguishing between customers and employees, thereby achieving automated access control and risk warning.

[0003] Object detection typically relies on deep learning models, which are trained on large amounts of labeled image data to automatically identify and locate target objects in images. During training, positive samples (images containing the target) are usually combined with negative samples (images without the target) to improve the model's recognition ability. However, in real-world financial scenarios, the number of positive samples can easily become too large or too small. This imbalance between positive and negative samples can cause the model to overlearn the features of the more numerous samples during training, while underlearning the features of the fewer numerous samples, thus affecting the accuracy of the model's detection. Summary of the Invention

[0004] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a training dataset processing method, apparatus, electronic device, and storage medium to make the various types of samples in the training dataset more balanced and improve the accuracy of model detection.

[0005] Firstly, this application provides a method for processing training datasets, including:

[0006] Feature recognition is performed on each target image in the target image dataset to obtain the distribution results of different feature categories; wherein, the target image dataset includes multiple target images containing human objects; the features include features used to distinguish the types of human objects;

[0007] Based on the distribution results, target element categories with a proportion lower than the target threshold are determined, and target images containing the target element categories are determined as sparse sample images.

[0008] Based on the features of the environment and people in the sparse sample image, a target text description is generated, and the target text description is input into the image generation model to obtain multiple augmented images corresponding to the sparse sample image;

[0009] A training dataset is constructed based on the target image dataset and multiple augmented images.

[0010] The training dataset processing method provided in this application embodiment identifies the distribution results of different element categories by performing element recognition on each target image in the target image dataset; wherein, the target image dataset includes multiple target images containing human objects; the elements include features used to distinguish the types of human objects; the target element categories with a proportion lower than a target threshold are determined according to the distribution results, and the target images containing the target element categories are determined as sparse sample images; target text descriptions are generated according to the features of the environment and human objects in the sparse sample images, and the target text descriptions are input into the image generation model to obtain multiple augmented images corresponding to the sparse sample images; a training dataset is constructed according to the target image dataset and the multiple augmented images. This application embodiment identifies the distribution of different element categories by performing element recognition on each image in the target image dataset. It then filters out sparse sample images with a proportion lower than the target threshold, accurately identifying samples of a minority class. Based on the environmental and human object features of the sparse sample images, it generates target text descriptions, which are then input into the image generation model to obtain multiple augmented images. This can specifically supplement the number of sparse samples, and the augmented images are consistent with the original sparse samples in terms of key features, enriching the quantity and diversity of minority class samples. This makes the samples of each class in the training dataset more balanced and improves the accuracy of model detection.

[0011] According to one embodiment of this application, the elements include clothing of the person object, and different element categories include different clothing categories; the person object types include employees and customers.

[0012] In this embodiment, in a financial scenario, employees and customers typically have different clothing characteristics. For example, employees may wear uniform work attire, while customers wear a variety of everyday clothes. By using clothing as a key feature to distinguish different types of people, the system can accurately identify the types of people corresponding to different clothing categories. By combining the target image dataset with augmented images to construct a training dataset, the proportion of various clothing samples corresponding to employees and customers in the training data becomes more balanced. This allows the model to fully learn the characteristics of different clothing categories, reducing recognition bias caused by too many or too few samples of a certain type of clothing, thereby more accurately distinguishing between employees and customers.

[0013] According to one embodiment of this application, the step of performing feature recognition on each target image in the target image dataset to obtain the distribution results of different feature categories includes:

[0014] The clothing category of the human figures in each target image is identified using a preset detection model;

[0015] The distribution results were obtained by counting the number of items in each clothing category.

[0016] In this embodiment, a preset detection model is used to analyze each image in the target image dataset to identify the clothing category of the people in the image. Key information can be extracted quickly and accurately, and the number of identified clothing categories can be counted to present the distribution of different clothing categories in the dataset. This allows for the accurate identification of which clothing categories have fewer samples, i.e., sparse samples.

[0017] According to one embodiment of this application, the features of the character object include the character object's actions and the character object's appearance;

[0018] The step of generating target text descriptions based on the features of the environment and people in the sparse sample images includes:

[0019] The environment in the sparse sample image is identified to obtain an environment description; and the actions and appearance of human objects in the sparse sample image are identified to obtain action descriptions and appearance descriptions.

[0020] A target text description is generated based on the environment description, the action description, and the appearance description.

[0021] In this embodiment, by identifying the environment in the sparse sample image and obtaining an environmental description, and by identifying the actions and appearance of human objects to obtain action and appearance descriptions, the key features of the sparse sample can be captured from different dimensions, so that the generated target text description accurately reflects the core information of the sparse sample image. Based on these multi-dimensional descriptions, target text is generated and input into the image generation model. The resulting augmented image not only retains the key elements such as clothing that distinguish employees from customers in the sparse sample image, but also maintains a high degree of consistency with the sparse sample image in terms of environment, action, and appearance, reducing the problem of feature distortion during the augmentation process.

[0022] According to one embodiment of this application, constructing a training dataset based on the target image dataset and a plurality of augmented images includes:

[0023] Read the pixel data of each target image in the target image dataset to determine the resolution of the human object in each target image;

[0024] Images containing human figures with a resolution lower than the target resolution are identified as anomalous sample images.

[0025] The abnormal sample images are repaired, and a training dataset is constructed based on the target image dataset after repairing the abnormal sample images and multiple augmented images.

[0026] In this embodiment, the resolution of the human figure is determined by reading the pixel data of each target image in the target image dataset, and images with a resolution lower than the target resolution are identified as abnormal sample images for repair. Then, the training dataset is constructed by combining the target dataset and the augmented image. This can further improve the quality of the training dataset and reduce the model learning bias caused by the blurring of human features in low-resolution images during model training.

[0027] According to one embodiment of this application, the repair of the abnormal sample image includes:

[0028] The abnormal sample images are repaired using an image restoration model;

[0029] The image restoration model is trained in the following manner:

[0030] Construct a sample set; the sample set includes multiple image pairs; each image pair includes an input image and a target image; the input image and the target image are images with the same content but different resolutions; the resolution of the target image is greater than the resolution of the input image;

[0031] A loss function is constructed based on category loss; the category loss represents the average difference between the restored image obtained by the image restoration model from the input image and the target image in multiple image pairs;

[0032] The sample set is input into the image restoration model, and the model parameters of the image restoration model are updated through backpropagation algorithm based on the loss function.

[0033] According to one embodiment of this application, the loss function is expressed as:

[0034]

[0035] Where L represents the loss function, and L1 represents the L1 paradigm difference between the repaired image and the corresponding target image. per This indicates the perceptual gap between the repaired image and the corresponding target image. L represents the feature difference between the input image and the corresponding repaired image. adv This represents the adversarial loss between the repaired image and the corresponding target image. L represents the contribution parameter against loss. cls This represents category loss.

[0036] In this embodiment, an image inpainting model is used to repair abnormal sample images. Combined with a specific model training method, the image inpainting model is trained through backpropagation using a sample set consisting of multiple image pairs containing the same content but with different resolutions, and a loss function constructed based on the class loss of the average difference between the repaired image and the target image. The class loss characterizes the average difference between the repaired image obtained by the image inpainting model and the target image in multiple image pairs. This makes the optimization objective of the loss function not limited to the repair accuracy of a single image pair, but pursues the consistency of the repaired image with the overall features of the same high-resolution target image. This can encourage the image inpainting model to learn the common features of the same type of image during the training process, thereby improving the repair effect of the image inpainting model on different categories of targets in abnormal sample images.

[0037] Secondly, this application provides a training dataset processing apparatus, the apparatus comprising:

[0038] The recognition module is used to perform feature recognition on each target image in the target image dataset to obtain the distribution results of different feature categories; wherein, the target image dataset includes multiple target images containing human objects; the features include features used to distinguish the types of human objects;

[0039] The determination module is used to determine the target element category with a proportion lower than the target threshold based on the distribution result, and to determine the target image containing the target element category as a sparse sample image.

[0040] The generation module is used to generate target text descriptions based on the features of the environment and human objects in the sparse sample images, and input the target text descriptions into the image generation model to obtain multiple augmented images corresponding to the sparse sample images;

[0041] A construction module is used to construct a training dataset based on the target image dataset and multiple augmented images.

[0042] The training dataset processing apparatus provided in this application embodiment identifies the distribution results of different element categories by performing element recognition on each target image in the target image dataset; wherein, the target image dataset includes multiple target images containing human objects; the elements include features used to distinguish the types of human objects; the target element categories with a proportion lower than a target threshold are determined according to the distribution results, and the target images containing the target element categories are determined as sparse sample images; target text descriptions are generated according to the features of the environment and human objects in the sparse sample images, and the target text descriptions are input into the image generation model to obtain multiple augmented images corresponding to the sparse sample images; a training dataset is constructed according to the target image dataset and the multiple augmented images. This application embodiment identifies the distribution of different element categories by performing element recognition on each image in the target image dataset. It then filters out sparse sample images with a proportion lower than the target threshold, accurately identifying samples of a minority class. Based on the environmental and human object features of the sparse sample images, it generates target text descriptions, which are then input into the image generation model to obtain multiple augmented images. This can specifically supplement the number of sparse samples, and the augmented images are consistent with the original sparse samples in terms of key features, enriching the quantity and diversity of minority class samples. This makes the samples of each class in the training dataset more balanced and improves the accuracy of model detection.

[0043] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the training dataset processing method as described in the first aspect above.

[0044] Fourthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training dataset processing method as described in the first aspect above.

[0045] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the training dataset processing method as described in the first aspect above.

[0046] The above-described one or more technical solutions in the embodiments of this application have at least one of the following technical effects:

[0047] The training dataset processing method provided in this application embodiment identifies the distribution results of different element categories by performing element recognition on each target image in the target image dataset; wherein, the target image dataset includes multiple target images containing human objects; the elements include features used to distinguish the types of human objects; the target element categories with a proportion lower than a target threshold are determined according to the distribution results, and the target images containing the target element categories are determined as sparse sample images; target text descriptions are generated according to the features of the environment and human objects in the sparse sample images, and the target text descriptions are input into the image generation model to obtain multiple augmented images corresponding to the sparse sample images; a training dataset is constructed according to the target image dataset and the multiple augmented images. This application embodiment identifies the distribution of different element categories by performing element recognition on each image in the target image dataset. It then filters out sparse sample images with a proportion lower than the target threshold, accurately identifying samples of a minority class. Based on the environmental and human object features of the sparse sample images, it generates target text descriptions, which are then input into the image generation model to obtain multiple augmented images. This can specifically supplement the number of sparse samples, and the augmented images are consistent with the original sparse samples in terms of key features, enriching the quantity and diversity of minority class samples. This makes the samples of each class in the training dataset more balanced and improves the accuracy of model detection.

[0048] Furthermore, in financial scenarios, employees and customers typically have different clothing characteristics. For example, employees may wear uniform work attire, while customers wear a variety of everyday clothes. By using clothing as a key feature to distinguish different types of people, it is possible to accurately identify the types of people corresponding to different clothing categories. By combining the target image dataset with augmented images to construct a training dataset, the proportion of various clothing samples corresponding to employees and customers in the training data is more balanced. This allows the model to fully learn the characteristics of different clothing categories, reducing recognition bias caused by too many or too few samples of a certain type of clothing, thereby more accurately distinguishing between employees and customers.

[0049] Furthermore, by analyzing each image in the target image dataset using a pre-set detection model, the clothing categories of the people in the images can be identified. Key information can be extracted quickly and accurately, and the number of identified clothing categories can be counted to present the distribution of different clothing categories in the dataset. This allows for the accurate identification of which clothing categories have fewer samples, i.e., sparse samples.

[0050] Furthermore, by identifying the environment in sparse sample images and obtaining environmental descriptions, and by recognizing the actions and appearances of human objects to obtain action and appearance descriptions, key features of sparse samples can be captured from different dimensions, enabling the generated target text description to accurately reflect the core information of the sparse sample images. Based on these multi-dimensional descriptions, target text is generated and input into the image generation model. The resulting augmented image not only retains key features such as clothing that distinguishes employees from customers in the sparse sample images, but also maintains a high degree of consistency with the sparse sample images in terms of environment, actions, and appearance, reducing the problem of feature distortion during the augmentation process.

[0051] Furthermore, by reading the pixel data of each target image in the target image dataset to determine the resolution of the human figure, and identifying images with lower resolutions as anomalous sample images for repair, and then combining the target dataset with the augmented images to construct a training dataset, the quality of the training dataset can be further improved, reducing the model learning bias caused by blurred human features in low-resolution images during model training.

[0052] Furthermore, by employing an image inpainting model to repair abnormal sample images, and combining it with a specific model training method, the image inpainting model is trained through backpropagation using a sample set consisting of multiple image pairs containing the same content but with different resolutions, and a loss function constructed based on the average difference between the repaired image and the target image. The category loss characterizes the average difference between the repaired image obtained by the image inpainting model and the target image in multiple image pairs. This ensures that the optimization objective of the loss function is not limited to the repair accuracy of a single image pair, but rather pursues the consistency of the repaired image with the overall features of the same high-resolution target image. This can encourage the image inpainting model to learn the common features of the same type of image during training, thereby improving the repair effect of the image inpainting model on different categories of targets in abnormal sample images.

[0053] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0054] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1This is a flowchart illustrating the training dataset processing method provided in an embodiment of this application;

[0056] Figure 2 This is a flowchart of sparse sample image augmentation based on an image generation model provided in an embodiment of this application;

[0057] Figure 3 This is a flowchart of abnormal sample image repair based on the CodeFormer model provided in the embodiments of this application;

[0058] Figure 4 This is a schematic diagram of the structure of the training dataset processing device provided in the embodiments of this application;

[0059] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0060] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0061] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0062] With the accelerated digital transformation of the financial industry, traditional manual service models are no longer sufficient to meet the demands for efficient, secure, and intelligent business operations. Target detection technology has become an indispensable key technology in financial scenarios. In locations such as bank branches, financial service halls, ATM areas, and intelligent security gates, personnel activities are closely related to business processes. Accurately identifying individuals in images through target detection, distinguishing between customers and employees, enables automated access control and risk warnings.

[0063] For example, in high-risk business areas, such as cash transport channels and important document storage rooms, access control can only be opened after employee identification, preventing customers from accidentally entering and causing security risks. When customers conduct large-amount transfers or open accounts, the system can identify the interaction between customers and employees in real time. Once abnormal actions or suspicious exchanges are detected, a risk warning mechanism is immediately triggered to effectively prevent fraud, illegal operations, and other behaviors, ensuring the safe and stable operation of financial services. As another example, in bank branches, the system can monitor the movement of people in the lobby in real time, automatically identify customers and employees entering the branch, and implement automated access control based on identity information. For customers, the system can guide them to the self-service area or counter to conduct business; for employees, it can record attendance and manage access to their work areas. Of course, it can also statistically analyze customer traffic to help financial institutions rationally allocate service resources and improve customer service experience.

[0064] Object detection technology typically relies on deep learning models, which learn the features of targets by training on large amounts of labeled image data, thereby enabling automatic identification and localization of target objects in images.

[0065] During training, positive samples (images containing the target) and negative samples (images without the target) are usually combined to improve the model's recognition ability. Positive samples act as "standard templates" for the model to learn, helping the model remember the features of the target, while negative samples serve as "negative examples," teaching the model to distinguish the target from other distracting objects.

[0066] Positive samples are images containing a target, which can be an employee or a customer. If a positive sample contains an image of an employee, then a negative sample contains an image of a customer, and vice versa.

[0067] In real-world financial scenarios, the number of positive samples is often unbalanced, either too high or too low. For example, during peak hours at bank branches, with a large number of customers, there will be a greater number of positive samples (images containing customers) and a relatively smaller number of negative samples (images containing employees). This imbalance between positive and negative samples can cause the model to overlearn the features of the more numerous samples during training, while underlearning the features of the fewer numerous samples. For instance, if the number of positive samples far exceeds the number of negative samples, the model may overfit to the features of positive samples and misclassify them when faced with negative samples, thus affecting the model's detection accuracy and generalization ability.

[0068] To address at least one of the aforementioned technical problems, this application provides a training dataset processing method, apparatus, and storage medium. The following, in conjunction with the accompanying drawings, provides a detailed description of the training dataset processing method, apparatus, and storage medium provided in this application through specific embodiments and application scenarios.

[0069] The training dataset processing method can be applied to the terminal, specifically executed by the hardware or software within the terminal.

[0070] The terminal includes, but is not limited to, portable communication devices such as mobile phones or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but rather a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0071] The following embodiments describe a terminal including a display and a touch-sensitive surface. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, mouse, and joystick.

[0072] The training dataset processing method provided in this application embodiment can be executed by an electronic device or a functional module or entity in an electronic device that can implement the method. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras, and wearable devices. The training dataset processing method provided in this application embodiment is described below using an electronic device as the execution subject.

[0073] like Figure 1 As shown, the training dataset processing method includes steps 110, 120, 130, and 140.

[0074] Step 110: Perform feature recognition on each target image in the target image dataset to obtain the distribution results of different feature categories; wherein, the target image dataset includes multiple target images containing human objects; the features include characteristics used to distinguish the types of human objects.

[0075] In this embodiment of the application, the target image dataset is an image collection containing multiple target images, and each target image contains a human figure.

[0076] In the financial sector, taking bank branch risk control as an example, the target images in the target image dataset mainly come from the bank's internal monitoring system. Banks replay historical videos recorded by the monitoring system and extract frames containing people or objects as target images. These target images cover scenes from different time periods and areas within the bank branch, such as customer service windows, self-service areas, and waiting areas. The people or objects in the target images can be customers conducting business or bank employees.

[0077] In the target image dataset, each target image contains multiple features, which are characteristics used to distinguish the types of people or objects in the image. These features can include the appearance, behavior, and location of the people or objects. For example, in a bank branch surveillance image, features could include clothing, actions, whether a person is wearing a mask, carrying items, and wearing an employee badge. Different representations of these features constitute different feature categories. For instance, clothing features can be divided into "wearing uniform" and "not wearing uniform," while action features can be divided into categories such as "walking normally," "standing," and "running."

[0078] In some embodiments, elements include clothing for human figures, with different element categories including different clothing categories; human figure types include employees and customers.

[0079] In this embodiment, in a financial scenario, employees and customers typically have different clothing characteristics. For example, employees may wear uniform work attire, while customers wear a variety of everyday clothes. By using clothing as a key feature to distinguish different types of people, the system can accurately identify the types of people corresponding to different clothing categories. By combining the target image dataset with augmented images to construct a training dataset, the proportion of various clothing samples corresponding to employees and customers in the training data becomes more balanced. This allows the model to fully learn the characteristics of different clothing categories, reducing recognition bias caused by too many or too few samples of a certain type of clothing, thereby more accurately distinguishing between employees and customers.

[0080] In some embodiments, image recognition technology and algorithms can be used to identify features in each target image within the target image dataset. Specifically, for each target image, image recognition tools can be used to analyze the human figures within it and extract features that can distinguish the type of human figure. For example, it can be used to identify whether a person in the target image is wearing work clothes, wearing a name tag, or whether they are inside or outside the counter.

[0081] In some embodiments, deep learning algorithms, such as Convolutional Neural Networks (CNNs), can also be used to extract and analyze features from the target image. First, the target image is input into a pre-trained neural network model, which automatically extracts feature information from the image. Then, a classifier categorizes the extracted features to determine whether specific elements exist in the target image and to which category these elements belong. For example, in surveillance images from a bank branch, the model can identify whether people in the image are wearing uniforms, thus distinguishing between employees and customers.

[0082] In this embodiment, by performing feature recognition on each target image in the target image dataset, the distribution results of different feature categories can be obtained. The distribution results reflect the frequency and proportion of each feature category in the target image dataset. For example, if the feature category "wearing uniform" accounts for 30% and the feature category "not wearing uniform" accounts for 70% in the target image dataset, then this distribution result can be obtained.

[0083] Step 120: Determine the target element category with a proportion lower than the target threshold based on the distribution results, and identify the target image containing the target element category as a sparse sample image.

[0084] After identifying the features of each target image in the target image dataset, the distribution results of different feature categories are obtained. These distribution results reflect the proportion of each feature category in the target image dataset. For example, in a bank branch surveillance image dataset, if the feature is clothing, the feature category can include "employees in uniform" and "customers not in uniform". Through statistical analysis, the specific proportion of each feature category can be determined, such as "wearing uniform" accounting for 20% and "not wearing uniform" accounting for 80%. If the feature is employee badge, the feature category can include "wearing employee badge" and "not wearing employee badge".

[0085] In training object detection models, a target threshold can be set to ensure balanced learning performance across different feature categories. The target threshold is a predefined proportion used to measure whether the percentage of a feature category is within a reasonable range. For example, the target threshold could be set to 30%. If the percentage of a feature category is lower than the target threshold, it indicates that the feature category is a sparse category in the dataset, meaning the number of samples is small.

[0086] Taking bank branch surveillance images as an example, assuming the target threshold is set at 20%, distribution analysis reveals that the proportion of the "not wearing uniform" element category is only 10%, lower than the target threshold. Therefore, the "not wearing uniform" element category can be identified as a sparse category, i.e., the target element category. Target images containing the target element category can be filtered from the target image dataset and identified as sparse sample images. Specifically, the target image dataset can be traversed, checking whether each target image contains the target element category. For example, for the "not wearing uniform" target element category, an element recognition algorithm can be used to check whether each target image contains an object not wearing a uniform. If it does, the target image is marked as a sparse sample image.

[0087] The purpose of identifying sparse sample images is to address the imbalance between positive and negative samples in the target image dataset. In financial scenarios, the number of samples for certain feature categories may be far fewer than those for other categories, leading to insufficient learning of sparse categories during model training. For example, in bank branch surveillance, there may be more samples of normal customer behavior, while samples of suspicious packages or abnormal behavior may be fewer. Without special processing of sparse sample images, the model may over-biasedly learn the features of common samples while ignoring the features of sparse samples, thus failing to accurately identify the target object corresponding to the sparse category in practical applications.

[0088] Step 130: Generate target text descriptions based on the features of the environment and people in the sparse sample images, and input the target text descriptions into the image generation model to obtain multiple augmented images corresponding to the sparse sample images.

[0089] In this embodiment of the application, the sparse sample images contain rich information. The environment may include background information of the image, such as the counter area, self-service area, lobby, etc. of a bank branch; while the features of the human object may include the person's appearance (such as clothing, hairstyle, whether wearing glasses, etc.), posture (such as standing, walking, bending over, etc.), actions (such as talking, operating equipment, raising hands, etc.), and the person's location (such as near the door, in front of the counter, etc.).

[0090] Taking bank branch surveillance images as an example, a sparse sample image might be a shot of a "person in uniform" in a self-service area. The environment can include background information such as the layout of the self-service area and the location of the equipment; the characteristics of the person can include the person's clothing, the shape and size of their package, and their posture.

[0091] Target text descriptions are textual representations of the environmental and human features in sparse sample images. The process of generating target text descriptions involves converting the extracted features into natural language descriptions so that image generation models can understand them and generate new images accordingly.

[0092] Natural language processing and computer vision technologies can be combined to generate target text descriptions. Specifically, image recognition tools can be used to perform deep analysis on sparse sample images, extracting various elements in the environment, such as scene layout, lighting conditions, and background objects, and capturing the features of human subjects, including clothing styles, identification badges, and postures. These extracted features are then organized in a logical order to transform them into a coherent and clear text description. In a bank branch scenario, for a sparse sample image containing "person in uniform," analysis might extract features such as "lobby," "self-service server," "wearing a suit and uniform," and "bending over," thereby generating a target text description such as "a person wearing a suit and uniform is standing in front of the second machine in the self-service area, operating the machine; the background is the bank's self-service area, surrounded by several self-service machines and some waiting people."

[0093] Image generation models are deep learning models that can generate corresponding images based on text descriptions. They are trained on large-scale datasets using deep learning techniques to learn the features and distribution patterns of images. After training, image generation models can transform text into visual content based on input conditions or descriptions.

[0094] In some embodiments, the image generation model can be a model based on architectures such as generative adversarial networks or variational autoencoders, which learns from a large amount of image and text pair data and has the ability to generate images based on text descriptions. The image generation model can also be a model based on diffusion models, such as Stable Diffusion and its derivatives, Midjourney, and the DALL E series (such as DALL-E, DALL-E2, etc.). The image generation model can also be based on other architectures, which are not limited in this embodiment.

[0095] In bank branch risk control scenarios, using sparse sample images as targets and inputting target text descriptions, the image generation model generates multiple augmented images that are similar to, but not identical to, the sparse sample images. For example, given the target text description "In the bank lobby at night, next to an ATM, a person in a dark suit and wearing a fluorescent badge stands near the glass partition of the teller counter," the image generation model might generate multiple augmented images with slightly different postures (such as turning around or raising a hand), slightly different lighting brightness, and different ATM display interfaces in the background. These augmented images retain the key features of the sparse sample images while increasing their diversity, effectively enriching the number of sparse sample images.

[0096] Step 140: Construct a training dataset based on the target image dataset and multiple augmented images.

[0097] The target image dataset contains target images covering various feature categories, but some feature categories may have a low proportion. Augmented images, on the other hand, are supplementary materials generated for target feature categories with an excessively low proportion (i.e., sparse sample images). Their purpose is to increase the proportion of these sparse feature categories in the dataset, thereby balancing the distribution of various features. For example, in a bank branch scenario, the target image dataset may contain a large number of images of "ordinary customers in uniform waiting in the waiting area," while images of "uniformed employees at the counter" are sparse sample images. The corresponding augmented images can supplement the number of this feature category, making the proportion of the two types of images in the dataset more reasonable.

[0098] The target image dataset and the augmented image dataset can be integrated into a single dataset to form a training dataset.

[0099] In some embodiments, data annotation and quality checks can also be performed on the training dataset. For example, data annotation can mark the location, category, and related attribute information of people in each image. For instance, in bank branch surveillance images, it can be marked whether each person is a "customer" or an "employee," whether they are carrying a package, and whether they are exhibiting abnormal behavior. For augmented images, although they are generated based on sparse sample images, accurate annotation is still required so that the model can learn correctly.

[0100] Quality control involves reviewing the images in the training dataset to ensure they meet quality requirements. Quality control can include checking image sharpness, resolution, accuracy of annotations, and whether the image content matches the actual scene. For example, if an augmented image has an incorrectly labeled person or a background that doesn't match a real bank branch scene, the augmented image can be regenerated or re-annotated. Through quality control, noise and errors in the training dataset can be reduced, improving the overall quality of the training dataset.

[0101] In some embodiments, the constructed training dataset can be used to train an object detection model. An object detection model is a computer vision technique used to automatically identify and locate target objects in images or videos. It can not only determine the presence of a target object in an image, but also accurately outline the target's location and classify it. Object detection models can be built based on deep learning techniques, such as convolutional neural networks. Common object detection model architectures include Faster R-CNN, YOLO (You Only Look Once), and SSD (Single Shot MultiBox Detector).

[0102] The constructed object detection model needs to be trained on a training dataset to better perform detection tasks. For example, in financial scenarios, object detection models can automatically identify people in images, distinguishing between customers and employees. For instance, in bank branch surveillance videos, object detection models can determine identity based on physical characteristics (such as whether they are wearing uniforms) or behavioral characteristics (such as whether they are operating at the counter).

[0103] The training dataset processing method provided in this application embodiment identifies the distribution results of different element categories by performing element recognition on each target image in the target image dataset. The target image dataset includes multiple target images containing human objects. The elements include features used to distinguish the types of human objects. The target element categories with a proportion lower than a target threshold are determined according to the distribution results, and the target images containing the target element categories are determined as sparse sample images. Target text descriptions are generated based on the features of the environment and human objects in the sparse sample images, and the target text descriptions are input into the image generation model to obtain multiple augmented images corresponding to the sparse sample images. A training dataset is constructed based on the target image dataset and the multiple augmented images. This application embodiment identifies the distribution of different element categories by performing element recognition on each image in the target image dataset. It then filters out sparse sample images with a proportion lower than the target threshold, accurately identifying samples of a minority class. Based on the environmental and human object features of the sparse sample images, it generates target text descriptions, which are then input into the image generation model to obtain multiple augmented images. This can specifically supplement the number of sparse samples, and the augmented images are consistent with the original sparse samples in terms of key features, enriching the quantity and diversity of minority class samples. This makes the samples of each class in the training dataset more balanced and improves the accuracy of model detection.

[0104] In some embodiments, feature recognition is performed on each target image in the target image dataset to obtain the distribution results of different feature categories, including:

[0105] The system identifies the clothing categories of human subjects in each target image using a pre-defined detection model.

[0106] The quantity of each clothing category was counted to obtain the distribution results.

[0107] In this embodiment, the detection model is a deep learning model specifically designed to identify the categories of clothing worn by people in images, and can be developed based on a convolutional neural network architecture. The detection model can automatically extract features from the image and classify the clothing of people into different categories, such as "uniform," "casual wear," and "formal wear."

[0108] Each image in the target image dataset can be sequentially input into the detection model. The detection model will analyze each image and identify the type of clothing worn by the people in the image. For example, in bank branch surveillance images, the detection model can identify whether a person is wearing a bank employee uniform or casual wear like a regular customer.

[0109] After identifying the clothing categories in the target images, it's necessary to count the number of people in each category. Specifically, a statistical table or data structure can be created to record the frequency of each clothing category in the target image dataset. For example, for the "uniform" category, the table would record the number of images identified as people wearing uniforms; for the "casual wear" category, the corresponding number would also be recorded. By iterating through the entire target image dataset and summing the identified clothing category results for each image into the statistical table, a complete distribution result can be obtained.

[0110] The distribution results can intuitively reflect the proportion of different clothing categories in the target image dataset. For example, if the statistical results show that the "uniform" category has a higher proportion and the "casual wear" category has a lower proportion, it means that in the bank branch surveillance images, there are relatively more employees and relatively fewer ordinary customers.

[0111] In this embodiment, a preset detection model is used to analyze each image in the target image dataset to identify the clothing category of the people in the image. Key information can be extracted quickly and accurately, and the number of identified clothing categories can be counted to present the distribution of different clothing categories in the dataset. This allows for the accurate identification of which clothing categories have fewer samples, i.e., sparse samples.

[0112] In some embodiments, the characteristics of a character object include the character object's actions and the character object's appearance;

[0113] Generate target text descriptions based on the features of environmental and human objects in sparse sample images, including:

[0114] Identify the environment in sparse sample images and obtain an environment description; and identify the actions and appearance of human objects in sparse sample images and obtain action descriptions and appearance descriptions.

[0115] Generate a target text description based on the environment description, action description, and appearance description.

[0116] In this embodiment, the characteristics of a character object include the character's actions and appearance. The actions of the character object can reflect the character object's behavioral state, while the appearance can include external expressions such as clothing and posture.

[0117] In this embodiment, a convolutional neural network can be used to extract the appearance features of people in sparse sample images, such as clothing color and style, and whether they are wearing masks or glasses. An action recognition model, such as one based on a temporal convolutional network or a recurrent neural network, can identify the person's actions, such as walking, running, standing, and gestures. For example, if a person is operating a self-service device in an image, the model will identify the person's action as "operating the device"; if the person is walking, the model will identify the person's action as "walking".

[0118] Environmental description is a summary of the background of an image, providing contextual information about the scene in which a person is situated. For example, in surveillance footage from a bank branch, the environment may include different areas such as the lobby, counters, and self-service areas.

[0119] To obtain an environmental description, scene recognition models can be used to analyze the image background. These models, trained on a large number of scene images, are able to identify key scene elements within an image. For example, by recognizing features such as counters, self-service machines, and seats in an image, it can be determined whether a person is in a bank lobby or a self-service area.

[0120] After obtaining the environmental description, action description, and appearance description, this information can be integrated into a complete target text description. For example, suppose in a bank branch's surveillance footage, a person wearing a dark suit is operating a machine in the self-service area, standing in front of a second machine. The target text description could be: "A man wearing a dark suit is operating the second machine in the self-service area." Such a description not only includes the person's actions and appearance information but also provides the environmental background, enabling the image generation model to generate more accurate and realistic augmented images based on this information.

[0121] Taking the DALL-E2 large model as an example, for instance, the image generation model uses this model. Figure 2 As shown, a target text description can be generated based on the environment description, action description, and appearance description. The target text description is then input into the DALL-E 2 large model to obtain the augmented image corresponding to the target text description.

[0122] The DALL-E2 large model is a text-to-image generation system based on two key technologies: CLIP (Contrastive Language-Image Pretraining) model and Diffusion Models. Its core principle is to achieve high-quality image generation through a two-stage generation architecture.

[0123] Phase 1: Prior Model: DALL-E2 first utilizes the CLIP model's text encoder and image encoder to extract features from the text and image respectively. The CLIP model learns semantic relationships between text and image by contrastive learning, mapping them to a shared feature space. The Prior model aims to generate corresponding latent image features based on the input text features. Text features extracted by the CLIP model's text encoder are then passed as conditional input to the Prior model. The Prior model progressively generates latent image representations using an autoregressive or diffusion model. During training, the CLIP model's image encoder transforms real images into image features, which serve as the ground truth to guide the Prior model's training.

[0124] The second stage: Decoder model: In the second stage, the decoder's task is to convert the latent features of the image generated by the Prior model into the actual high-resolution image. DALL-E2 uses diffusion models as the basis for the decoder. Diffusion models generate an image that conforms to the input description from random noise through progressive denoising, thereby achieving the function of augmenting sparse sample images.

[0125] In this embodiment, by identifying the environment in the sparse sample image and obtaining an environmental description, and by identifying the actions and appearance of human objects to obtain action and appearance descriptions, the key features of the sparse sample can be captured from different dimensions, so that the generated target text description accurately reflects the core information of the sparse sample image. Based on these multi-dimensional descriptions, target text is generated and input into the image generation model. The resulting augmented image not only retains the key elements such as clothing that distinguish employees from customers in the sparse sample image, but also maintains a high degree of consistency with the sparse sample image in terms of environment, action, and appearance, reducing the problem of feature distortion during the augmentation process.

[0126] In some embodiments, a training dataset is constructed based on a target image dataset and multiple augmented images, including:

[0127] Read the pixel data of each target image in the target image dataset and determine the resolution of the human figure in each target image;

[0128] Images containing human figures with a resolution lower than the target resolution are identified as anomalous sample images.

[0129] The abnormal sample images are repaired, and a training dataset is constructed based on the target image dataset after repairing the abnormal sample images and multiple augmented images.

[0130] In this embodiment, for the target image dataset, some target images may be blurry due to equipment, lighting, or other factors during image acquisition, which is detrimental to subsequent model training. The pixel data of each target image can be read to analyze the details and quality of the target image. Pixel data is a fundamental component of an image, containing the color and brightness information of each pixel.

[0131] Resolution refers to the number of pixels contained within a unit length. After reading pixel data, the number of pixels in the area where a person or object is located in the target image can be counted to obtain the resolution of the person or object. For example, if the number of pixels in the area where the person or object is located is 1000×800, the resolution of the person or object can be represented as 1000×800.

[0132] In this embodiment, the resolution of the human figure can be compared with a preset target resolution. The target resolution is a quality standard used to ensure that the human figure in the image is sharp enough so that the model can learn effectively during subsequent training. For example, in a bank branch risk control scenario, it is necessary to accurately distinguish between customers and employees, which requires clear identification of clothing. Therefore, the target resolution is usually set at a level that can clearly present these details. For example, the target resolution can be set to 32×32, or other values.

[0133] If the resolution of a person in a target image is lower than the target resolution, then that target image will be marked as an anomalous sample image. Anomalous sample images may have blurry figures due to factors such as shooting distance being too far, poor camera quality, or image compression.

[0134] In this embodiment, abnormal sample images can be repaired to improve image quality, enabling the repaired abnormal sample images to meet or approach the target resolution requirements. In some embodiments, image enhancement techniques, super-resolution reconstruction techniques, etc., can be used to repair abnormal sample images, making the originally blurry abnormal sample images clearer, allowing the model to better learn the features of people during subsequent training.

[0135] In this embodiment, the resolution of the human figure is determined by reading the pixel data of each target image in the target image dataset, and images with a resolution lower than the target resolution are identified as abnormal sample images for repair. Then, the training dataset is constructed by combining the target dataset and the augmented image. This can further improve the quality of the training dataset and reduce the model learning bias caused by the blurring of human features in low-resolution images during model training.

[0136] In some embodiments, repairing abnormal sample images includes:

[0137] Anomaly sample images are repaired using an image restoration model;

[0138] The image restoration model was trained in the following way:

[0139] Construct a sample set; the sample set includes multiple image pairs; each image pair includes an input image and a target image; the input image and the target image are images with the same content but different resolutions; the resolution of the target image is greater than the resolution of the input image;

[0140] A loss function is constructed based on category loss; category loss represents the average difference between the restored image obtained by the image inpainting model from the input image and the target image in multiple image pairs;

[0141] The sample set is input into the image restoration model, and the model parameters of the image restoration model are updated through backpropagation algorithm based on the loss function.

[0142] In this embodiment, the image inpainting model is a deep learning model used to restore missing or damaged parts of an image. By learning the features and structure of a large number of complete images, the image inpainting model can automatically fill in blank areas, remove noise, and repair scratches or other damaged parts, making the restored image visually natural and coherent while preserving as much information as possible from the original image. In some embodiments, the image inpainting model used may be CodeFormer, LaMa, Stable Diffusion Inpainting, MAT (Mask-Aware Transformer), etc., and this application embodiment does not limit this to any particular model.

[0143] The CodeFormer model, used as an example, is a blurry image restoration model based on VQ-GAN (Vector Quantized Generative Adversarial Networks) and Transformer. It is specifically designed to address anomalies such as blurring of small objects due to image pixel size limitations. The core design idea is to change the paradigm of traditional restoration tasks by leveraging the pre-trained VQ-GAN discrete codebook space to transform the blurry image restoration task into a code sequence prediction task. This transformation significantly reduces the uncertainty of mapping relationships in the restoration task. Simultaneously, the VQ-GAN codebook prior provides rich details of the blurred image for the restoration process. Combined with the global modeling capabilities of the Transformer, this further enhances the model's robustness to severely degraded image restoration, ultimately resulting in a more realistic and natural restored blurry image.

[0144] like Figure 3As shown, the training process of the CodeFormer model is divided into two stages: encoding and quantization, and repair and super-resolution.

[0145] The encoding and quantization stages are fundamental to the training of the CodeFormer model. In this stage, the CodeFormer model uses an encoder to convert a high-quality image into corresponding feature representations, and then uses a codebook to quantize these features. The quantized features are then input into a decoder, which reconstructs the original high-quality image. This process not only achieves effective compression of image information, but more importantly, it preserves the key features of the image during compression, laying a data foundation for subsequent tasks such as blurred image restoration and super-resolution reconstruction.

[0146] The inpainting and super-resolution stages are the core components of the CodeFormer model's image restoration. In this stage, the CodeFormer model first receives the low-quality image to be restored and extracts image features using a trained encoder. Then, it uses the Transformer module to model the extracted global features, predicting the index corresponding to each feature. Finally, with the help of a fixed pre-trained decoder, the quantized features are reconstructed into a high-quality image. Throughout this process, the Transformer module captures global relationships within the image, thereby reducing restoration problems caused by local degradation and ensuring that the restored image achieves ideal results in both overall coherence and detail accuracy.

[0147] During the application phase of the CodeFormer model, small images of people or objects can be cropped from the abnormal sample images. These small images are then input into the trained CodeFormer model, which outputs repaired high-resolution small images. These repaired high-resolution small images are then pasted back into their corresponding positions in the abnormal sample images to obtain the repaired abnormal sample images.

[0148] The specific training process is described below. First, a sample set can be constructed, which includes multiple image pairs. Each image pair includes an input image and a target image. The content of the input image and the target image is the same, but their resolutions differ, and the resolution of the target image is higher than that of the input image.

[0149] For example, a clear image of a person can be selected from high-resolution surveillance images of a bank branch as the target image. This image can then be downsampled using professional image processing tools, such as reducing the number of pixels or the image size, to obtain a corresponding low-resolution image as the input image. It should be noted that the image pairs in the sample set can cover various scenarios within the bank branch, such as lighting conditions at different times of day and the poses of people in different locations, ensuring that the image restoration model maintains the stability of its restoration effect even in complex environments.

[0150] Category loss refers to the average difference between the restored image output by the image inpainting model and the target image in the sample set. This difference can include errors at the pixel level or inconsistencies at the feature level. For example, when comparing the restored image and the target image, a feature extraction network can be used to analyze whether the key elements of the person (such as name tag identification, clothing texture) are consistent. These differences are quantified and averaged to obtain the category loss. The role of the loss function is to transform the category loss into an optimizable objective for the image inpainting model. That is, the greater the difference between the restored image and the target image, the higher the loss value. The loss can be reduced by adjusting the parameters of the image inpainting model. When the loss value is reduced to a preset range, it indicates that the image inpainting model can achieve good restoration from low resolution to high resolution.

[0151] When a sample set is input into an image restoration model, the restoration effect on the input image is often less than ideal in the early stages of training. For example, the restored image may have distorted details or blurred edges, resulting in a high loss value for the calculated loss function. The backpropagation algorithm calculates the impact of the loss value on the parameters of each layer (i.e., the gradient value) along the network structure of the image restoration model, from the output layer to the input layer, and then adjusts the parameters according to the gradient direction. This training process is iterated repeatedly until the difference between the restored image and the target image output by the image restoration model stabilizes at a low level, and the loss value no longer decreases significantly.

[0152] In some embodiments, the loss function is expressed as:

[0153]

[0154] Where L represents the loss function, and L1 represents the L1 paradigm difference between the repaired image and the corresponding target image. per This indicates the perceptual gap between the repaired image and the corresponding target image. L represents the feature difference between the input image and the corresponding repaired image. adv This represents the adversarial loss between the repaired image and the corresponding target image. L represents the contribution parameter against loss. cls This represents category loss.

[0155] In this embodiment, an image inpainting model is used to repair abnormal sample images. Combined with a specific model training method, the image inpainting model is trained through backpropagation using a sample set consisting of multiple image pairs containing the same content but with different resolutions, and a loss function constructed based on the class loss of the average difference between the repaired image and the target image. The class loss characterizes the average difference between the repaired image obtained by the image inpainting model and the target image in multiple image pairs. This makes the optimization objective of the loss function not limited to the repair accuracy of a single image pair, but pursues the consistency of the repaired image with the overall features of the same high-resolution target image. This can encourage the image inpainting model to learn the common features of the same type of image during the training process, thereby improving the repair effect of the image inpainting model on different categories of targets in abnormal sample images.

[0156] The training dataset processing method provided in this application can be executed by a training dataset processing device. This application uses an example of a training dataset processing device executing the training dataset processing method to illustrate the training dataset processing device provided in this application.

[0157] This application also provides a training dataset processing apparatus.

[0158] like Figure 4 As shown, the training dataset processing device includes:

[0159] The recognition module 410 is used to perform feature recognition on each target image in the target image dataset to obtain the distribution results of different feature categories; wherein, the target image dataset includes multiple target images containing human objects; the features include features used to distinguish the types of human objects;

[0160] The determination module 420 is used to determine the target element category with a proportion lower than the target threshold based on the distribution results, and to determine the target image containing the target element category as a sparse sample image.

[0161] The generation module 430 is used to generate target text descriptions based on the features of the environment and human objects in the sparse sample images, and input the target text descriptions into the image generation model to obtain multiple augmented images corresponding to the sparse sample images;

[0162] Module 440 is used to construct a training dataset based on the target image dataset and multiple augmented images.

[0163] The training dataset processing apparatus provided in this application embodiment obtains the distribution results of different element categories by performing element recognition on each target image in the target image dataset; wherein, the target image dataset includes multiple target images containing human objects; the elements include features used to distinguish the types of human objects; the target element categories with a proportion lower than a target threshold are determined according to the distribution results, and the target images containing the target element categories are determined as sparse sample images; target text descriptions are generated based on the features of the environment and human objects in the sparse sample images, and the target text descriptions are input into the image generation model to obtain multiple augmented images corresponding to the sparse sample images; a training dataset is constructed based on the target image dataset and the multiple augmented images. This application embodiment identifies the distribution of different element categories by performing element recognition on each image in the target image dataset. It then filters out sparse sample images with a proportion lower than the target threshold, accurately identifying samples of a minority class. Based on the environmental and human object features of the sparse sample images, it generates target text descriptions, which are then input into the image generation model to obtain multiple augmented images. This can specifically supplement the number of sparse samples, and the augmented images are consistent with the original sparse samples in terms of key features, enriching the quantity and diversity of minority class samples. This makes the samples of each class in the training dataset more balanced and improves the accuracy of model detection.

[0164] In some embodiments, the identification module 410 is further configured to:

[0165] The system identifies the clothing categories of human subjects in each target image using a pre-defined detection model.

[0166] The quantity of each clothing category was counted to obtain the distribution results.

[0167] In some embodiments, the generation module 430 is further configured to:

[0168] Identify the environment in sparse sample images and obtain an environment description; and identify the actions and appearance of human objects in sparse sample images and obtain action descriptions and appearance descriptions.

[0169] Generate a target text description based on the environment description, action description, and appearance description.

[0170] In some embodiments, the construction module 440 is further configured to:

[0171] Read the pixel data of each target image in the target image dataset and determine the resolution of the human figure in each target image;

[0172] Images containing human figures with a resolution lower than the target resolution are identified as anomalous sample images.

[0173] The abnormal sample images are repaired, and a training dataset is constructed based on the target image dataset after repairing the abnormal sample images and multiple augmented images.

[0174] In some embodiments, the construction module 440 is further configured to:

[0175] Anomaly sample images are repaired using an image restoration model;

[0176] The image restoration model was trained in the following way:

[0177] Construct a sample set; the sample set includes multiple image pairs; each image pair includes an input image and a target image; the input image and the target image are images with the same content but different resolutions; the resolution of the target image is greater than the resolution of the input image;

[0178] A loss function is constructed based on category loss; category loss represents the average difference between the restored image obtained by the image inpainting model from the input image and the target image in multiple image pairs;

[0179] The sample set is input into the image restoration model, and the model parameters of the image restoration model are updated through backpropagation algorithm based on the loss function.

[0180] The training dataset processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0181] The training dataset processing device in this application embodiment can be a device with an operating system. This operating system can be a Microsoft (Windows) operating system, an Android operating system, an iOS operating system, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0182] In some embodiments, such as Figure 5 As shown, this application embodiment also provides an electronic device 500, including a processor 501, a memory 502, and a computer program stored in the memory 502 and executable on the processor 501. When the program is executed by the processor 501, it implements the various processes of the above-described training dataset processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0183] It should be noted that the electronic devices in the embodiments of this application include the aforementioned mobile electronic devices and non-mobile electronic devices.

[0184] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described training dataset processing method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0185] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0186] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described training dataset processing method.

[0187] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0188] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described training dataset processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0189] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0190] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0191] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0192] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0193] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0194] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for processing training datasets, characterized in that, include: Feature recognition is performed on each target image in the target image dataset to obtain the distribution results of different feature categories; wherein, the target image dataset includes multiple target images containing human objects; the features include features used to distinguish the types of human objects; Based on the distribution results, target element categories with a proportion lower than the target threshold are determined, and target images containing the target element categories are determined as sparse sample images. Based on the features of the environment and people in the sparse sample image, a target text description is generated, and the target text description is input into the image generation model to obtain multiple augmented images corresponding to the sparse sample image; A training dataset is constructed based on the target image dataset and multiple augmented images.

2. The method according to claim 1, characterized in that, The elements include the clothing of the characters, and different element categories include different clothing categories; the types of characters include employees and customers.

3. The method according to claim 2, characterized in that, The step of performing feature recognition on each target image in the target image dataset to obtain the distribution results of different feature categories includes: The clothing category of the human figures in each target image is identified using a preset detection model; The distribution results were obtained by counting the number of items in each clothing category.

4. The method according to claim 1, characterized in that, The characteristics of the character object include the character object's actions and the character object's appearance; The step of generating target text descriptions based on the features of the environment and people in the sparse sample images includes: The environment in the sparse sample image is identified to obtain an environment description; and the actions and appearance of human objects in the sparse sample image are identified to obtain action descriptions and appearance descriptions. A target text description is generated based on the environment description, the action description, and the appearance description.

5. The method according to claim 1, characterized in that, The step of constructing a training dataset based on the target image dataset and multiple augmented images includes: Read the pixel data of each target image in the target image dataset to determine the resolution of the human object in each target image; Images containing human figures with a resolution lower than the target resolution are identified as anomalous sample images. The abnormal sample images are repaired, and a training dataset is constructed based on the target image dataset after repairing the abnormal sample images and multiple augmented images.

6. The method according to claim 5, characterized in that, The repair of the abnormal sample image includes: The abnormal sample images are repaired using an image restoration model; The image restoration model is trained in the following manner: Construct a sample set; the sample set includes multiple image pairs; each image pair includes an input image and a target image; the input image and the target image are images with the same content but different resolutions; the resolution of the target image is greater than the resolution of the input image; A loss function is constructed based on category loss; the category loss represents the average difference between the restored image obtained by the image restoration model from the input image and the target image in multiple image pairs; The sample set is input into the image restoration model, and the model parameters of the image restoration model are updated through backpropagation algorithm based on the loss function.

7. The method according to claim 6, characterized in that, The loss function is expressed as: Where L represents the loss function, and L1 represents the L1 paradigm difference between the repaired image and the corresponding target image. per This indicates the perceptual gap between the repaired image and the corresponding target image. L represents the feature difference between the input image and the corresponding repaired image. adv This represents the adversarial loss between the repaired image and the corresponding target image. L represents the contribution parameter against loss. cls This represents category loss.

8. A training dataset processing apparatus, characterized in that, include: The recognition module is used to perform feature recognition on each target image in the target image dataset to obtain the distribution results of different feature categories; wherein, the target image dataset includes multiple target images containing human objects; the features include features used to distinguish the types of human objects; The determination module is used to determine the target element category with a proportion lower than the target threshold based on the distribution result, and to determine the target image containing the target element category as a sparse sample image. The generation module is used to generate target text descriptions based on the features of the environment and human objects in the sparse sample images, and input the target text descriptions into the image generation model to obtain multiple augmented images corresponding to the sparse sample images; A construction module is used to construct a training dataset based on the target image dataset and multiple augmented images.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

Images