Method and device for processing training sample set

Abstract

The invention relates to a method for processing a training sample set, a device for processing the training sample set, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring a training sample from a first training sample set, and determining a representation vector of the training sample; based on the representation vector of the training sample, determining a linear combination of concept vectors corresponding to the training sample; determining a sampling probability of the training sample based on a linear combination of the concept vectors corresponding to the training sample; and determining a second training sample set based on the sampling probability of the training samples. According to the method and the device, the second training sample set without deviation or with little deviation can be generated from the first training sample set with potential deviation, and the quality of the training sample set for training various neural network models is remarkably improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence services, and more specifically to a method for processing training sample sets, an apparatus for processing training sample sets, electronic devices, computer-readable storage media, and computer program products. Background Technology

[0002] With the research and advancements in deep learning and machine learning, neural network models are being researched and applied in various fields, such as social networking, e-commerce, drug discovery, smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, neural network models will be applied in more fields and play an increasingly important role.

[0003] In the fields of deep learning and machine learning, the quality of the training sample set plays a crucial role in the performance and fairness of neural network models. Potential decision shortcuts may exist within the training sample set, which could cause the neural network model to learn biased decision rules, thus affecting the fairness and accuracy of the neural network model in practical applications.

[0004] Specifically, in the training sample set, certain decision rules unrelated to the task of the neural network model may adversely affect the model during training. These "decision rules" are also known as "decision shortcuts." If the neural network model learns these "decision shortcuts" instead of correct judgment criteria, the trained neural network model will produce biased decision results during inference.

[0005] Therefore, it is necessary to improve the existing schemes for constructing training sample sets in order to effectively identify and eliminate various potential biases that may exist in the training sample sets, so that the neural network model can learn fair and unbiased decision rules. Summary of the Invention

[0006] This disclosure provides a method for processing a training sample set, an apparatus for processing a training sample set, an electronic device, a computer-readable storage medium, and a computer program product.

[0007] This disclosure provides a method for processing a training sample set, comprising: obtaining training samples from a first training sample set and determining the representation vector of the training samples; determining a linear combination of concept vectors corresponding to the training samples based on the representation vectors of the training samples, wherein the linear combination of concept vectors is a weighted sum of multiple concept vectors semantically associated with the training samples; determining the sampling probability of the training samples based on the linear combination of concept vectors corresponding to the training samples; and determining a second training sample set based on the sampling probability of the training samples.

[0008] This disclosure provides an apparatus for processing a training sample set, the apparatus comprising: a first module configured to: acquire training samples from a first training sample set and determine the representation vector of the training samples; a second module configured to: determine a linear combination of concept vectors corresponding to the training samples based on the representation vectors of the training samples, wherein the linear combination of concept vectors is a weighted sum of multiple concept vectors semantically associated with the training samples; a third module configured to: determine the sampling probability of the training samples based on the linear combination of concept vectors corresponding to the training samples; and a fourth module configured to: determine a second training sample set based on the sampling probability of the training samples.

[0009] This disclosure provides an electronic device, including: one or more processors; and one or more memories, wherein the memories store a computer-executable program, and when the processor executes the computer-executable program, the above-described method is performed.

[0010] This disclosure provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the above-described method.

[0011] According to another aspect of this disclosure, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable medium and executes the computer instructions, causing the computer device to perform the methods provided in the foregoing aspects or various alternative implementations of the foregoing aspects.

[0012] This disclosure enables the generation of an unbiased or minimally biased second training sample set from a potentially biased first training sample set, significantly improving the quality of training sample sets used to train various neural network models. Specifically, training a neural network using the second training sample set determined by this disclosure allows the neural network model to learn fair and unbiased decision rules, avoiding the occurrence of "decision shortcuts." Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. The accompanying drawings in the following description are merely exemplary embodiments of this disclosure.

[0014] Figure 1 This is an example schematic diagram illustrating a scenario according to an embodiment of the present disclosure.

[0015] Figure 2 A schematic diagram of a neural network model according to an embodiment of the present disclosure is shown.

[0016] Figure 3 A schematic diagram of training samples in a training sample set according to an embodiment of the present disclosure is shown.

[0017] Figure 4 A flowchart of a method for processing a training sample set according to an embodiment of the present disclosure is shown.

[0018] Figure 5 A schematic diagram of an exemplary concept dictionary according to embodiments of the present disclosure is shown.

[0019] Figure 6 A schematic diagram of a neural network model according to an embodiment of the present disclosure is shown.

[0020] Figure 7 A schematic diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0021] Figure 8 A schematic diagram of the architecture of an exemplary computing device according to an embodiment of the present disclosure is shown.

[0022] Figure 9 A schematic diagram of a storage medium according to an embodiment of the present disclosure is shown. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0024] In this specification and accompanying drawings, operations and elements that are substantially the same or similar are indicated by the same or similar reference numerals, and repeated descriptions of these operations and elements are omitted. Furthermore, in the description of this disclosure, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance or order.

[0025] To facilitate the description of this disclosure, the following concepts related to this disclosure are introduced.

[0026] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0027] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, large language neural network models, operating / interaction systems, and mechatronics. Among these, large language neural network models, also known as large neural network models or basic neural network models, can be widely applied to downstream tasks across various AI fields after fine-tuning. AI software technologies mainly include computer vision, speech processing, natural language processing, and machine learning / deep learning. With the research and advancement of AI technology, it is being researched and applied in numerous fields, such as smart homes, wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, digital twins, virtual humans, robots, AI-generated content (AIGC), conversational interaction, smart healthcare, smart customer service, and game AI. It is believed that with further technological development, AI will be applied in even more fields and play an increasingly important role.

[0028] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning. Large language neural network models represent the latest development in deep learning, integrating all of these techniques.

[0029] Optionally, the neural network models used in embodiments of this disclosure as described below can all be artificial intelligence neural network models, especially artificial intelligence-based neural network models. Typically, artificial intelligence-based neural network models are implemented as acyclic graphs, where neurons are arranged in different layers. Generally, a neural network model includes an input layer and an output layer, separated by at least one hidden layer. The hidden layer transforms the input received by the input layer into a representation useful for generating the output in the output layer. Nodes are fully connected to nodes in adjacent layers via edges, and there are no edges between nodes within each layer. Data received at nodes in the input layer of the neural network is propagated to nodes in the output layer via any of the hidden layers, activation layers, pooling layers, convolutional layers, etc. The input and output of the neural network model can take various forms, and this disclosure does not limit them.

[0030] The solutions provided in this disclosure involve technologies such as artificial intelligence and / or machine learning, which are specifically illustrated through the following embodiments.

[0031] First, refer to Figure 1 This disclosure describes application scenarios of methods for processing data according to embodiments of the present disclosure, as well as corresponding apparatuses, etc. Figure 1 A schematic diagram of an application scenario 100 according to an embodiment of the present disclosure is shown, wherein a server 110 and a plurality of terminals 120 are schematically illustrated.

[0032] The neural network model of this disclosure can be integrated into various electronic devices, for example. Figure 1 The neural network model can be integrated into any electronic device in server 110 and multiple terminals 120. For example, the neural network model can be integrated into terminal 120. Terminal 120 can be a mobile phone, tablet, laptop, desktop computer, personal computer (PC), smart speaker, or smartwatch, but is not limited to these. Alternatively, the neural network model can also be integrated into server 110. Server 110 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals and servers can be directly or indirectly connected via wired or wireless communication, and this disclosure does not impose any limitations.

[0033] It is understood that the apparatus for inference using the neural network model of this disclosure can be a terminal, a server, or a system composed of a terminal and a server. The data processing method of this disclosure can be executed on a terminal, on a server, or jointly by a terminal and a server.

[0034] The artificial intelligence neural network model provided in this disclosure can also relate to artificial intelligence cloud services in the field of cloud technology. Cloud technology refers to a hosting technology that unifies hardware, software, network, and other resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology applied based on the cloud computing business model. It can form resource pools, be used on demand, and is flexible and convenient. Cloud computing technology will become an important support. The backend services of technical network systems require a large amount of computing and storage resources, such as video websites, image websites, drug research websites, and many more portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to a backend system for logical processing. Data at different levels will be processed separately, and various industry data will require strong system support, which can only be achieved through cloud computing.

[0035] It is worth noting that both the terminal 110 and server 120 according to the embodiments of this disclosure adhere to data protection principles, respect users' data rights, and safeguard users' data security and privacy. The terminal 110 and server 120 according to the embodiments of this disclosure will clearly inform users of the purpose, method, and scope of collecting, using, storing, transmitting, and deleting user data, and obtain users' consent. The terminal 110 and server 120 according to the embodiments of this disclosure will take reasonable technical and management measures to prevent user data from being leaked, tampered with, damaged, or lost. The providers of the terminal 110 and server 120 according to the embodiments of this disclosure will regularly review and update user data, and promptly delete expired or useless data. Furthermore, cloud service providers using the embodiments of this disclosure respect users' rights to data access, correction, deletion, withdrawal of consent, complaints, and claims, and provide convenient channels and procedures to enable users to effectively exercise these rights.

[0036] Furthermore, the process of data analysis using artificial intelligence technology in terminal 120 or server 110 is conducted based on the principles of legality, rationality, and transparency. The data collected and processed by the artificial intelligence neural network model according to embodiments of this disclosure is relevant, necessary, and appropriate for the prediction purpose, and does not contain any personally identifiable or sensitive information. The neural network model according to embodiments of this disclosure employs appropriate technologies and organizational measures to protect the security and integrity of data, preventing unauthorized access, use, or disclosure.

[0037] The artificial intelligence-based neural network model according to embodiments of this disclosure will comply with relevant data protection regulations and ethical principles. This neural network model is trained on a large amount of anonymized and de-identified data, and does not infringe on the privacy rights of any individual or group. The artificial intelligence neural network model has also undergone rigorous testing and evaluation to ensure that its output results are accurate and reliable, and will not cause any misleading or discriminatory behavior. The artificial intelligence neural network model is designed solely to improve service quality and customer satisfaction and will not be used for any illegal or unethical purposes. Furthermore, the neural network model will be regularly reviewed and updated to adapt to changes in the data environment and legal regulations.

[0038] "Decision shortcuts" refer to the phenomenon where, during the training of a machine learning neural network model, the model may unintentionally learn patterns or decision rules unrelated to the target and rely on these patterns or rules to make decisions. This phenomenon typically occurs when the neural network model tries to find the simplest way to achieve its performance metrics, ignoring complex or subtle factors in the real world. This learning based on biases in the training sample set rather than true features leads to incorrect decisions in real-world applications. The emergence of "decision shortcuts" indicates that the neural network model may not truly understand the essential characteristics of the target object, but rather relies on irrelevant or even misleading features in the training sample set. Such biases not only reduce the accuracy of the neural network model in real-world applications but may also lead to inconsistent performance in different scenarios. This erroneous decision shortcut severely impacts the reliability and impartiality of the neural network model.

[0039] The formation of "decision shortcuts" usually stems from defects or biases in the training data itself. For example, in animal recognition tasks, if a large number of images containing "crabs" have a beach background, while images containing "spiders" have a forest background, then the neural network model may use "the presence of a beach" as a shortcut feature to determine "whether it is a spider," rather than truly learning the essential features that distinguish "crabs" and "spiders."

[0040] Furthermore, "decision shortcuts" not only affect the generalization ability of neural network models, but may also exacerbate the inherent biases and unfairness of neural network models. For example, if certain groups in the training sample set are associated with some irrelevant features for some reason, then the neural network model will learn these irrelevant features and make biased judgments about that group.

[0041] Therefore, eliminating decision-making shortcuts in the training sample set and ensuring that the neural network model learns the correct decision-making logic is crucial. To address the bias problem in neural network models, researchers have attempted to balance the dataset using resampling techniques, such as assigning greater weight to samples that the neural network model struggles to classify correctly. However, this approach primarily focuses on the sample's label information, neglecting other important attributes such as background and other objects in the scene.

[0042] While this coarse-grained approach can reduce bias in neural network models to some extent, it still cannot completely eliminate bias in the training sample set because it does not deeply analyze and process the intrinsic attributes and contextual information of the samples. Furthermore, it may introduce new problems or ignore subtle differences between samples.

[0043] Specifically, relying solely on label information for resampling leads to the following problems. First, differences between samples within the same category are ignored. For example, images containing the word "dog" may differ significantly in background, pose, etc. Even after resampling, these differences remain, and label information cannot eliminate irrelevant features stemming from differences in background or pose. Second, similarities between samples from different categories are also ignored. For instance, "dog" and "wolf" are quite similar in appearance, but relying solely on labels cannot effectively enable neural network models to learn their differences. Furthermore, label-based coarse-grained resampling may overlook a significant amount of contextual information. For example, in face recognition tasks, the background environment of a face image greatly influences recognition, a fact that label information cannot reflect.

[0044] The embodiments disclosed herein aim to obtain a higher-quality training sample set by performing more granular analysis and processing, utilizing concept decomposition technology to conduct deeper attribute analysis and balancing of training samples. These embodiments can more effectively reduce bias in neural network models during training, improving the fairness and accuracy of the neural network models.

[0045] Specifically, embodiments of this disclosure provide a method for processing a training sample set, comprising: obtaining training samples from a first training sample set and determining the representation vector of the training samples; determining a linear combination of concept vectors corresponding to the training samples based on the representation vectors of the training samples, wherein the linear combination of concept vectors is a weighted sum of multiple concept vectors semantically associated with the training samples; determining the sampling probability of the training samples based on the linear combination of concept vectors corresponding to the training samples; and determining a second training sample set based on the sampling probability of the training samples.

[0046] The following reference Figures 2 to 9 Methods and apparatus for processing training sample sets according to embodiments of the present disclosure are described.

[0047] Figure 2 A schematic diagram of a neural network model according to an embodiment of the present disclosure is shown. Figure 3 A schematic diagram of training samples in a training sample set according to an embodiment of the present disclosure is shown.

[0048] For example, the neural network model according to embodiments of this disclosure can be used to perform an image classification task. In an image classification task, the goal of the neural network model is to correctly classify and predict the type of input image data based on its visual features. Figure 2 As shown, this neural network model can distinguish whether the input image data includes the sun or the moon.

[0049] Figure 2 The neural network model shown can consist of multiple convolutional neural network layers, each extracting feature vectors from the original image pixels. Each convolutional neural network layer can be composed of multiple convolutional and pooling layers stacked alternately to effectively capture the local patterns and spatial hierarchy of the image. The extracted feature map representations are typically tiled and fed into fully connected layers. The role of the fully connected layers is to perform highly non-linear combinations and transformations of the features from the previous layers to learn higher-dimensional feature representations, ultimately mapping to an output dimension equal to the number of classes, with each dimension corresponding to a confidence score for one class. For example, Figure 2 It shows two categories, "sun" and "moon".

[0050] To enhance the discriminative ability and generalization performance of neural network models, the neural network models according to embodiments of this disclosure may also employ regularization techniques such as batch normalization and dropout to alleviate the gradient vanishing problem in neural network models.

[0051] In order to make Figure 2 The neural network model in the text can perform image classification tasks efficiently and accurately, and can be used... Figure 3The first training sample set shown is used for training. Figure 2 The neural network model in the example. The first training sample set includes the first training sample and the second training sample.

[0052] Figure 3 Examples of a first training sample and a second training sample are given. Both the first and second training samples include sample image data and corresponding text labels. The first and second training samples correspond to different categories. For example, the sample image data of the first training sample includes data about the sun, while the sample image data of the second training sample includes data about the moon. The text labels of the first training sample also indicate the category to which the sample image data belongs. During training, the neural network model will attempt to learn how to effectively distinguish the visual features of "sun" and "moon" based on the text labels corresponding to the sample image data. However, the sample labels in the first training sample set may be imbalanced, possibly including a large amount of data about the moon and fewer data related to the sun. Also, the sun in the first training sample set may appear simultaneously with clouds, while the moon may not appear simultaneously with clouds. Therefore, the neural network model may mistakenly identify the appearance of "clouds" as a key feature for recognizing the sun and moon. As mentioned above, directly using the first training sample set for training... Figure 2 Neural network models may lead to the phenomenon of "decision shortcuts".

[0053] Therefore, it is necessary to utilize references. Figure 4 The described method 40 generates a second training sample set based on a first training sample set. This second training sample set corrects for biases in the first training sample set to avoid the phenomenon of "decision shortcuts." Specifically, in the second training sample set, the difference in the number of training samples for the sun and the moon may be reduced, while more training samples excluding cloud patterns will be sampled.

[0054] Figure 4 A flowchart of a method 40 for processing a training sample set according to an embodiment of the present disclosure is shown.

[0055] Method 40 can be used on terminal devices or servers (such as...) Figure 1 The method is executed at the terminal 120 or server 110. Method 40 includes the following operations S401 to S404. Of course, method 40 may also include more operations (e.g., operation S405) or fewer operations, and this disclosure is not limited thereto.

[0056] In operation S401, training samples are obtained from the first training sample set, and the representation vector of the training samples is determined.

[0057] Optionally, one or more training samples can be selected from the first training sample set for subsequent processing by traversal or random selection.

[0058] Optionally, if the task of the neural network model to be trained is an image-related task (e.g., image recognition, image classification, image enhancement, text-to-image, image-to-text, etc.), then the training samples used to train the neural network model may be multimodal. For example, each training sample in the first training sample set may include both sample image data and the text labels corresponding to the sample image data. Of course, this disclosure is not limited thereto.

[0059] Optionally, when the training samples include sample image data and text labels, operation S401 may further include: determining an image representation vector corresponding to the sample image data based on the sample image data; determining a text representation vector corresponding to the text label based on the text label; and determining a representation vector of the training sample based on the image representation vector corresponding to the sample image data and the text representation vector corresponding to the text label; wherein the image representation vector corresponding to the sample image data and the text representation vector corresponding to the text label correspond to the same semantic space. This disclosure is not limited thereto.

[0060] Suppose the first training sample set can be used as S = {(I i ,T i )} i＝1...N Let I be used to represent this. i T represents the sample image data of the i-th training sample. i Indicates with I i The corresponding text labels, where N is the total number of samples in the first training sample set. Then, assuming that in operation S401, the i-th training sample S is selected... i Of course, this disclosure is not limited to this.

[0061] Optionally, the representation vector of the training samples may include at least one of image representation vectors and text representation vectors. Specifically, any encoder can be used to convert sample image data into image representation vectors, or any encoder can be used to convert text labels into text representation vectors. The representation vector of the training samples may include only image representation vectors, or it may be a concatenation of image representation vectors and text representation vectors. This disclosure does not impose any limitations on the encoder.

[0062] The following uses the image encoder and text encoder of the pre-trained CLIP (Contrastive Language–Image Pre-training) neural network model as examples to illustrate the process of obtaining the image representation vector and text representation vector of the training samples.

[0063] The CLIP neural network model consists of two main parts: an image encoder f img and text encoder f text These two image encoders and text encoders use contrastive learning to process sample image data I. i and text label I i They are mapped to the same semantic space so that image representation vectors and text representation vectors can be directly compared.

[0064] Optionally, in operation S401, the sample image data I i It can be preprocessed to a fixed size and normalized to fit the image encoder f. img Input requirements. The image encoder f of the CLIP neural network model. img It can be a convolutional neural network (CNN) or a vision transformer (ViT). Image encoder f img The preprocessed image input can be converted into a fixed-size high-dimensional vector v. Ii High-dimensional vector v Ii This is the image representation vector. This process can be denoted as v. Ii =f img (I i ).

[0065] Similarly, in operation S401, the text label T i The text can first be segmented and converted into word embeddings, which represent the feature vector of each word. CLIP's text encoder f... text Based on a Transformer architecture, word embedding sequences can be input into the transformer. Through multi-layered self-attention mechanisms and feedforward neural networks, a high-dimensional vector v of fixed size can be output. Ti High-dimensional vector v Ti This is the text representation vector. This process can be denoted as v. Ti =f txt (T i ).

[0066] In operation S402, based on the representation vector of the training sample, a linear combination of concept vectors corresponding to the training sample is determined, wherein the linear combination of concept vectors is a weighted sum of multiple concept vectors semantically associated with the training sample.

[0067] Optionally, the concept vectors corresponding to the training samples can be understood as vectors abstracted from the training samples that can represent key data attributes. These concept vectors are semantically related to the training samples. For example, when processing image data, the concept vectors may be specific types of visual features related to the content presented by the image, such as semantics, edges, color distribution, texture, etc., corresponding to parts of the image. Each training sample may correspond to the result of a linear combination of multiple concept vectors. This disclosure is not limited thereto.

[0068] Optionally, the linear combination of the concept vectors is a new vector formed by weighted summation of multiple concept vectors, used to characterize the comprehensive semantics of the training sample and the importance of each concept to the training sample. For example, suppose the semantics of the training sample is "king," and the semantics of the three concept vectors are known to be "queen," "man," and "woman," respectively. Then, the vector corresponding to the training sample "king" can be represented as the "queen" vector minus the "woman" vector plus the "man" vector. Of course, this disclosure is not limited to this.

[0069] Optionally, the representation vectors of the training samples can be projected onto the concept space using a concept decomposition scheme, thereby decomposing the representation vectors of the training samples into linear combinations of different concept vectors. Concept decomposition can break down high-dimensional feature vectors into simpler, easier-to-understand, and easier-to-process low-dimensional feature vectors, thus capturing the latent patterns and information of the training samples. Concept decomposition of the representation vectors of training samples can use techniques such as dimensionality reduction, feature extraction, and matrix factorization.

[0070] For example, principal component analysis (PCA) can be used to project the representation vectors of the training samples onto a few principal components using linear transformations, preserving the main variability of the representation vectors while reducing dimensionality. Alternatively, nonnegative matrix factorization (NMF) can be used to decompose the data matrix into the product of two nonnegative matrices to determine latent patterns and information in the training samples. This disclosure is not limited thereto.

[0071] Suppose that the CLIP neural network model is used in operation S401 to obtain the representation vectors of the training samples. The CLIP neural network model is linear in the concept space, which is closely related to the linear representation assumption. Specifically, the CLIP neural network model, through contrastive learning, enables images and text to have the property of being linearly representable in the encoded vector space. This linear representation assumption means that at the semantic concept level, the representation vectors of images and text can also be mapped and aligned with each other through simple linear transformations. This assumption is consistent with the principle of concept decomposition, which enables the decomposition of complex high-dimensional data into simpler, easier-to-understand, and easier-to-process low-dimensional representations.

[0072] To illustrate the details of operation S402 more concisely within the context of concept decomposition, the image encoder f is then... img =f and text encoder f text =g. Simultaneously, let the sample image data I... i =x img And text label T i =x Txt At this point, the image representation vector v Ii =f img (I i )=f(x img ), and the text representation vector v Ti =f txt (T i )=g(x txt ).

[0073] Optionally, assuming that the representation vector of the training sample includes the image representation vector corresponding to the sample image data, operation S402 further includes: obtaining a concept dictionary, the concept dictionary including one or more concept vectors; and optimizing the image representation vector corresponding to the sample image data by linear regression to transform it into a product of a sparse vector and the concept dictionary, the product of the sparse vector and the concept dictionary being a linear combination of the concept vectors corresponding to the training sample.

[0074] After conceptual decomposition, the image representation vector v Ii It can be represented as The text representation vector v Ti It can be represented as Among them, symbols This indicates the composition of functions. In mathematics and computation, a composite function represents the computation of a result by using the output of one function as the input of another.

[0075] ω represents the concept space, which consists of multiple concept vectors and contains sample image data I. i and text label T i The semantic information in the context. The concept space refers to an abstract space composed of various related concept vectors, where each concept vector represents an abstract, distinguishable concept or category. ε represents the noise space composed of non-semantic factors, containing noise and other irrelevant information. The function h... img and h txt It is a mapping function from latent variables to data, describing how to generate specific sample image data I from the set (ω,ε) of the concept space and noise space. i and text label T i .

[0076] Furthermore, the CLIP encoder is linear in the concept space ω and independent of the noise space ε (i.e., non-semantic factors in the image, such as angle, illumination, etc.). Specifically, although and In normal circumstances, this might be non-linear, but in the case of a training sample set for an image classification task, and In fact, in the concept space ω, it can be represented as a linear function of multiple concept vectors, and and It is independent of the noise space ε. Therefore, the image representation vector v Ii =f img (I i It can be optimized by linear regression into a linear combination of one or more concept vectors in the concept space ω.

[0077] Based on the above principles, in one embodiment of this disclosure, operation S402 may further include: obtaining a concept dictionary, the concept dictionary including one or more concept vectors; determining the concept vector corresponding to the training sample based on the text representation vector corresponding to the text label; and converting the image representation vector corresponding to the sample image data into a product of a sparse vector and the concept dictionary based on the concept vector corresponding to the training sample, the product of the sparse vector and the concept dictionary being a linear combination of the concept vectors corresponding to the training sample.

[0078] Suppose the above process can be represented as f(x) img ) = CN. Where, N∈R k+l , C∈R d×(k+l) N is a sparse vector representing the weights of each concept vector. C is the vector representing the text v. Ti set v T The concept dictionary is obtained from multiple word embeddings, where d is the dimension of the vector, and k and l are the number of background and feature concepts, respectively. This indicates that, through linear regression, the image representation f(x) img This can be represented as the product of a sparse vector N and a concept dictionary C. This formulation provides a formal, mathematically based method for identifying and extracting concepts, allowing the use of the linear properties of the CLIP neural network model to decompose complex image and text labels into more easily understood and processed semantic concepts. Further references will follow. Figure 5 An example of the concept dictionary C is described in detail here, but will not be repeated here.

[0079] In addition to using linear regression optimization, considering the strong correlation between the image representation vectors corresponding to the sample image data and the text representation vectors corresponding to the text labels, the following operations can also be performed in operation S402: obtaining a concept dictionary, the concept dictionary including one or more concept vectors; determining the concept vectors corresponding to the training samples based on the text representation vectors corresponding to the text labels; and transforming the image representation vectors corresponding to the sample image data into a product of a sparse vector and the concept dictionary based on the concept vectors corresponding to the training samples, the product of the sparse vector and the concept dictionary being a linear combination of the concept vectors corresponding to the training samples.

[0080] Specifically, the semantic information of the text label may be consistent with the semantic information of the sample image data. Therefore, it can be directly derived from the text representation vector v. Ti Extract the concept vectors corresponding to the training samples. Specifically, this can be done directly based on the text representation vector v. Ti The distances between the multiple word embeddings and the concept vectors in the concept dictionary are used to determine the concept vectors closest to these word embeddings as the concept vectors corresponding to the training samples. This disclosure is not limited to this. After obtaining the concept vectors corresponding to the training samples, the image representation vectors corresponding to the sample image data can be easily written as the product of a sparse vector and the concept dictionary, thereby obtaining a linear combination of the concept vectors corresponding to the training samples.

[0081] In operation S403, the sampling probability of the training sample is determined based on the linear combination of the concept vectors corresponding to the training sample.

[0082] Optionally, the sampling probability of the training samples refers to the probability that the training samples are selected for training the neural network model. Optionally, the sampling probability of the training samples can be determined using a balanced sampling technique. The balanced sampling technique aims to ensure that the samples in the training sample set remain balanced across all concept dimensions. This means that each concept or category should be relatively balanced in proportion when actually applied to the training of the neural network model, so as to ensure that the neural network model does not favor any particular concept or category during training.

[0083] For example, optionally, operation S403 may include: obtaining a concept dictionary corresponding to the first training sample set, the concept dictionary including one or more concept vectors; determining a balance index of the first training sample set based on the concept dictionary, the balance index indicating the distribution balance of each concept in the concept dictionary in the representation vectors of the training samples of the first training sample set; and determining the sampling probability of the training samples based on the balance index of the first training sample set. Of course, this disclosure is not limited thereto.

[0084] Suppose that the concept dictionary C can be represented as a set C = {C1, C2, ..., C} m}, where each C i As a feature representation of a (key) concept, C is therefore further... i This is called a concept vector, where m is the total number of concept vectors.

[0085] Specifically, each training sample S in the first training sample set i eigenvector x i These concepts can be represented or approximated by a linear combination of their vectors, i.e.: x i =∑ j＝1…m W ij ·Cj+∈ i Among them, x i It can refer to the above image representation vector v Ii , or it can refer to the text representation vector v above. Ti Of course, x i It can also refer to the image representation vector v Ii and text representation vector v Ti The weighted sum or concatenation of x. The following uses x as an example. i =v Ii This is used as an example for illustration, but this disclosure is not limited to this.

[0086] Among them, W ij This indicates that concept Cj exists in sample x. i The corresponding weights, ∈ i This is the noise term, representing influencing factors other than the identified concepts. The above linear combination illustrates that for each sample x... i All of these can be viewed as being generated by a linear combination of multiple conceptual features. Weight W ij This indicates the importance or contribution of each concept in the training samples.

[0087] Suppose that a balanced sampling technique is used in operation S402. During this balanced sampling process, a balance index is defined to measure the distribution balance of concepts in the training sample set. Optionally, it is necessary to calculate the average weight of each concept across all samples. Specifically, let... Representing concept C i The average weight across all samples, then Where N is the total number of samples.

[0088] Next, balanced sampling can be achieved by adjusting the sampling probability of each sample. For each training sample x i The probability p of it being selected i It can be determined based on its deviation from the average concept weight. Specifically, the training sample xi sampling probability p i This reflects the representativeness of the sample in the conceptual dimension. Samples that deviate more from the average conceptual weight have a higher probability of being selected, ensuring that the neural network model can learn more from these samples with special features. For example, training samples x can be... i sampling probability p i Set as Wherein, the sampling probability p is defined in this way i This ensures that samples with significant deviations in certain conceptual dimensions are selected with a higher probability. Therefore, embodiments of this disclosure achieve a balanced distribution of concepts in the training sample set, ensuring greater sample diversity and reducing bias caused by uneven concept distribution, thereby improving the generalization ability and overall performance of the neural network model.

[0089] In operation S404, a second training sample set is determined based on the sampling probability of the training samples.

[0090] For example, in operation S404, the number K of training samples in the second training sample set can be determined first. Then, for example, a weighted random sampling method can optionally be used, based on the sampling probability p of each sample. i Sample selection is then performed. Specifically, for each training sample, the sampling probability can be determined by following operations S401 to S403. Then, in operation S404, a random number between 0 and 1 is generated for each training sample, and its sampling probability p is determined. i The random numbers are weighted. Then, the K training samples with the highest weighted random values ​​are selected to form the training sample set.

[0091] For example, 0 or 1 can be randomly generated based on the sampling probability of the training sample. If 0 is generated, it means that the training sample will not be used in this training process, that is, the training sample will not be added to the second training sample set. If 1 is generated, it means that the training sample can be used in this training process, and the training sample will be added to the second training sample set.

[0092] Of course, the second training sample set can also be determined based on the sampling probabilities of the training samples in other ways. For example, the sampling probabilities corresponding to each training sample can be sorted, and the top K training samples can be selected as the second training sample set. This disclosure is not limited thereto.

[0093] Furthermore, in an optional embodiment of this disclosure, new, diversified training samples can be generated in operation S404 by identifying and combining key concepts in the data. These training samples are then added to a second training sample set. For example, in operation S404, the following can be performed: determining at least one key concept based on a balance index of the first training sample set; generating expanded training samples based on a linear combination of concept vectors corresponding to the at least one key concept; and adding the expanded training samples to the second training sample set. Specifically, the key concepts can be those with average weights across all samples. A relatively high-level concept. Alternatively, a key concept may be a concept that appears less frequently in the first training sample set. This disclosure is not limited to this.

[0094] Optionally, embodiments of this disclosure may further include optional operation S405. In operation S405, a neural network model is trained based on the second training sample set.

[0095] Optionally, training the neural network model includes: obtaining a concept dictionary corresponding to the first training sample set, the concept dictionary including one or more concept vectors; determining a loss function corresponding to the neural network model based on an empirical loss and a regularization term; and training the neural network model using the second training sample set based on the loss function; wherein the empirical loss indicates the difference between the predicted values ​​of the neural network model for training samples in the second training sample set and the true values ​​of the training samples in the second training sample set; and the regularization term indicates the degree of alignment between the model parameters of the neural network model and at least one concept vector in the concept dictionary.

[0096] For example, suppose the neural network model is a parameterized function, where f θ This represents the parameters of the neural network model. The second training sample set D ′ ={(x1) ′ ,y1 ′ ),(x2 ′ ,y2 ′ ),…,(x ′ N ,y ′ N )}, where x i ′ The image representation vector v of the sample Ii y i ′ It is the corresponding text label.

[0097] The goal of training the neural network model is to minimize the empirical loss, typically by optimizing the following loss function. This is achieved by... Here, l is the loss function, used to measure the predicted value f of the neural network model. θ (x i ′ ) and the real label y i ′ The difference between them. In this formula, f θ (x i ′ ) is a neural network model f θ For training sample x i ′ The predicted value. l(f) θ (x i ′ ),y i ′ ) represents the predicted value f of the neural network model. θ (x i ′ ) and the real label y i ′ The losses between them.

[0098] By minimizing the loss function L(θ), the optimal neural network model parameters f can be found in operation S405. θ This makes the neural network model f θ Second training sample set D ′ The prediction error is minimized on the target. The form of the loss function L can be chosen according to the specific task; for example, mean squared error (MSE) or cross-entropy loss can be used.

[0099] Optionally, to further improve the efficiency of the neural network model in utilizing concept information, a regularization term can be introduced to encourage the neural network model to rely more on concepts when making predictions. This regularization term can be defined as the degree of alignment between the neural network model parameters θ and the concept vector C, that is, Therefore, the overall optimization objective can be transformed into minimizing the objective function: Here, λ is a regularization coefficient used to balance the importance of the empirical loss and the regularization term. The objective function consists of two parts: the empirical loss L(θ) and the regularization term R(θ,C). The empirical loss L(θ) ensures that the neural network model fits the training data well, while the regularization term R(θ,C) encourages the neural network model parameters θ to be closer to the concept vector, thereby improving the efficiency of the neural network model in utilizing concept information. By adjusting the regularization coefficient λ, the trade-off between the empirical loss and the regularization term can be controlled, achieving better neural network model performance and generalization ability.

[0100] It's important to note that "encouraging the neural network model parameters θ to be closer to the concept vectors" does not directly equate to replacing the values ​​of the neural network model parameters with the values ​​of the "concept vectors." Rather, it refers to the parameter representations that the neural network model should attempt to learn during training, which are associated with specific concepts in the data. Specifically, by designing an objective function that includes a regularization term, the parameter tuning of the neural network model during training can be made to tend to capture these concepts and ensure that the neural network model parameters effectively reflect these concepts. This allows the neural network model to rely on its understanding of these concepts rather than on irrelevant features or noise in the data when making predictions.

[0101] Therefore, this disclosure can generate an unbiased or minimally biased second training sample set from a first training sample set with potential bias, and use the second training sample set to train a neural network, enabling the neural network model to learn fair and unbiased decision rules and avoiding the occurrence of "decision shortcuts".

[0102] Specifically, embodiments of this disclosure perform more granular analysis and processing of the training sample set, utilizing concept decomposition technology to conduct deeper attribute analysis and balancing of the training samples. This more effectively reduces bias in the training process of neural network models, improving their fairness and accuracy. Furthermore, embodiments of this disclosure provide new perspectives and tools for ethical and fairness research in the field of machine learning. By deeply analyzing biases in data and models, researchers can better understand the root causes of these biases and develop new methods to reduce or eliminate them, promoting the healthy and sustainable development of machine learning.

[0103] Furthermore, in one embodiment of this disclosure, the interpretability of the neural network model can be improved by utilizing concept decomposition techniques and regularization terms. This embodiment, through regularization terms, links the decision-making process of the neural network model to human-understandable concepts as much as possible, making it easier for researchers and users to understand the behavior of the neural network model and to identify and correct biases in its decision-making. This is crucial for improving the credibility and acceptance of neural network models in sensitive and critical applications.

[0104] Furthermore, in one embodiment of this disclosure, concept decomposition techniques can be used for data augmentation and synthesis to generate new and diverse training samples by identifying and combining key concepts in the data. This not only increases the size and diversity of the second training sample set but also helps the model learn more robust feature representations and improves its generalization ability to unseen samples.

[0105] Furthermore, the embodiments disclosed herein are particularly suitable for adversarial machine learning scenarios. Specifically, in adversarial machine learning, attackers typically deceive neural network models by introducing subtle perturbations into the input data. Concept decomposition and balanced sampling techniques can be used to enhance the robustness of neural network models against such attacks. By ensuring that the neural network model takes into account the diversity and complexity of the data during training, the over-reliance of the neural network model on a specific data distribution can be reduced, thereby improving its ability to identify adversarial examples.

[0106] To further explain some details of the concept dictionary, please refer to the following... Figure 5 Taking the training of a neural network model to enable it to handle tasks based on image recognition of spiders as an example, a hypothetical concept dictionary is constructed to further illustrate operation S402.

[0107] Figure 5 A schematic diagram of an exemplary concept dictionary according to embodiments of the present disclosure is shown. Figure 5 The example shown is intended to demonstrate how to extract concept vectors from a set of text labels Ti using a CLIP neural network model and determine a concept dictionary based on the extracted concept vectors.

[0108] Figure 5 The process of constructing the concept dictionary shown can be summarized as follows: based on each training sample in the first training sample set, determine the text label of each training sample; extract at least one keyword from the text label of each training sample; determine the concept vector corresponding to the at least one keyword based on the at least one keyword; and determine the concept dictionary based on the concept vector corresponding to the at least one keyword. Of course, this disclosure is not limited thereto.

[0109] First, obtain the first training sample set, in which the training samples S i Includes sample image data of spiders I i and related text tags T i Sample image data I i The background can be diverse, such as a beach, forest, or home environment. The goal of operation S402 is based on the text label T. i Identification and extraction of sample image data I i Related concepts.

[0110] like Figure 5 As shown on the left, the text label T iIt could be something like, "On the beach, you can clearly see a long-legged spider moving slowly among the sand grains in the sunlight," "In a dense forest, a web-weaving spider is spinning an exquisite web between two trees, with sunlight shining brightly through the leaves onto the web," or "In a corner of the room, a spider is slowly crawling in the dim light."

[0111] Keywords are identified by analyzing the text labels of the training samples in the first training sample set. This process can be performed using any keyword extraction algorithm, such as TF-IDF (Term Frequency-Inverse Document Frequency) or word vector models (such as Word2Vec, GloVe, BERT). The identified keywords can then be categorized into specific categories using a predefined category dictionary or clustering methods. A predefined category dictionary is used as an example here.

[0112] Specifically, a category dictionary can be predefined, which can classify keywords into three main categories: background concepts, spider feature concepts, and image quality concepts.

[0113] For example, the concept of background refers to the environment or background elements in an image or scene, which are usually related to the subject or target object, but not necessarily directly related to it. Background concepts include image backgrounds such as beaches, forests, and home environments, etc. These environments can influence the visual characteristics of spiders and the neural network model's perception of them. For instance, a beach refers to an image with a sandy background, which may not be directly related to spiders, but since crabs are commonly found on beaches, it may cause the neural network model to learn incorrect decision paths. A forest refers to an environment with trees and vegetation as the background, a common ecological environment for spiders, which helps the neural network model correctly learn spider characteristics. A home environment refers to an image taken inside a home, which may include various furniture and objects as a background, and its impact on the neural network model's recognition of spiders is complex and variable.

[0114] The concept of spider characteristics encompasses both morphological and behavioral features. Morphological characteristics include a spider's physical features, such as leg length, body size, and coloration; these features are crucial for spider identification and classification. For example, a long-legged spider may belong to a specific species. Behavioral characteristics describe a spider's behavior, such as whether it spins webs and its crawling method. These behavioral characteristics not only reflect a spider's habits but also contribute to a deeper understanding of spider species and their characteristics.

[0115] Image quality encompasses both image sharpness and lighting conditions. Sharpness refers to the clarity of an image, specifically the degree to which the outlines and details of objects are displayed. High-resolution images help neural network models learn the detailed features of spiders more accurately because higher sharpness makes it easier to capture and analyze the spider's morphological and structural features. Lighting conditions, on the other hand, describe the lighting environment in which the image was taken, including natural light and indoor light. The visible features of spiders may differ under different lighting conditions; for example, the color and texture of a spider may appear different in strong sunlight or in a dim environment. Therefore, lighting conditions are also a crucial factor affecting the neural network model's ability to recognize spider features.

[0116] Figure 5 The right side shows an example of a concept dictionary, which is illustrated by... Figure 5 Text labels are generated on the left. This embodiment of the disclosure uses a linear regression method based on the CLIP neural network model to generate text labels from sample image data I. i and related text tags T i The vector representations of the aforementioned concepts are extracted. For example, a vector representation of each concept in the concept dictionary can be generated using a text encoder of the CLIP model. Of course, other methods can also be used to generate the vector representation of each concept in the concept dictionary, and this disclosure is not limited thereto.

[0117] Next, the vector representations of all concepts can be combined into a matrix C, where each column corresponds to the vector representation of a concept. Thus, the concept dictionary can be successfully constructed.

[0118] As described above, the concept dictionary helps the embodiments of this disclosure to more accurately understand the features and environmental background of spider images, providing a foundation for subsequent data analysis and applications. This method of constructing a concept dictionary is not only applicable to the specific scenario of spiders, but can also be extended to the recognition and understanding of other objects or scenes, providing an important reference for research and applications in the fields of machine learning and computer vision.

[0119] Next, taking a fully connected network (Feedforward Neural Network) as an example of the neural network to be trained, some optional details of the embodiments of this disclosure will be further introduced.

[0120] Figure 6 A schematic diagram of a neural network model according to an embodiment of the present disclosure is shown. Although in Figure 6 In this document, a fully connected network (Feedforward Neural Network) model is used as an example to illustrate some optional details of the embodiments of this disclosure. However, those skilled in the art should understand that this disclosure is not limited thereto.

[0121] exist Figure 6 In this embodiment of the present disclosure, the neural network model performs tasks such as feature extraction, image recognition, object detection, image segmentation, and image enhancement of image data.

[0122] Assume that the neural network model of this embodiment includes multiple fully connected layers, each of which can be defined as: y = f(Wx + b), where W is the weight matrix, x is the input vector, b is the bias term, and f is the activation function.

[0123] Optionally, the output of each fully connected layer in the neural network model may be associated with a vector representation of a concept. For example, all or some of the fully connected layers in the neural network model may be designed to capture specific features of image data. For instance, some fully connected layers may be used to capture the concept of "edges" in an image, while others may be used to capture the concept of "color".

[0124] To ensure that specific fully connected layers have sufficient data to learn representations of specific concepts during neural network model training, a reference layer can be used. Figures 1 to 4 Method 40, described in detail, determines a second training sample set and uses the second training sample set to train the neural network model. Through adjustments made by operations S401 to S404 in method 40, the training samples in the second training sample set may correspond to multiple concepts, and the distribution of each concept is roughly balanced.

[0125] Suppose that the training samples in the second training sample set correspond to multiple concepts, and the concept dictionary is denoted as concept dictionary C. During the training of the neural network model, the value of the weight matrix W can be adjusted using regularization techniques so that certain rows or columns of W are aligned with the feature vectors of one or more concepts in concept dictionary C.

[0126] Specifically, this can be achieved through a regularization term R(W,C) in the loss function used to train the neural network model, for example, R(W,C)=∥WC∥ 2 The core of this training process is to guide the neural network model to consider concepts directly relevant to the task during learning, rather than simply fitting the data, through regularization. This improves the model's sensitivity to concepts, reduces bias, and enhances its generalization ability by aligning the model's parameters with the concept vectors. The trained neural network model can then make more accurate and fair predictions when faced with new, unseen samples because it learns a broad range of meaningful features, rather than special cases or random noise from the training data.

[0127] It is worth noting that although the embodiments of this disclosure are primarily described in the field of image recognition and processing, they do not actually limit the application areas. For example, in Natural Language Processing (NLP), the embodiments of this disclosure can also be used to identify biases in text labels and reduce biases during model training through balanced sampling. Similarly, in recommender systems, the diversity and fairness of recommendations can be improved by identifying potential biases in user preferences and performing balanced processing.

[0128] Furthermore, embodiments of this disclosure also provide an apparatus for processing a training sample set, comprising: a first module configured to: acquire training samples from a first training sample set and determine the representation vector of the training samples; a second module configured to: determine a linear combination of concept vectors corresponding to the training samples based on the representation vectors of the training samples; a third module configured to: determine the sampling probability of the training samples based on the linear combination of concept vectors corresponding to the training samples; and a fourth module configured to: determine a second training sample set based on the sampling probability of the training samples.

[0129] According to another aspect of this disclosure, an electronic device is also provided for implementing the methods according to embodiments of this disclosure. Figure 7 A schematic diagram of an electronic device 2000 according to an embodiment of the present disclosure is shown.

[0130] like Figure 7 As shown, the electronic device 2000 may include one or more processors 2010 and one or more memories 2020. The memories 2020 store computer-readable code that, when executed by the one or more processors 2010, can perform the methods described above.

[0131] The processor in this disclosure embodiment can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, operations, and logic block diagrams disclosed in this disclosure embodiment. The general-purpose processor can be a microprocessor or any conventional processor, and can be based on an x86 architecture or an ARM architecture.

[0132] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0133] For example, the method or apparatus according to embodiments of this disclosure can also be used by means of Figure 8 The architecture of the computing device 3000 shown is used for implementation. For example... Figure 8 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage devices in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the methods provided in this disclosure, as well as program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 8 The architecture shown is merely exemplary and can be omitted as needed when implementing different devices. Figure 8 One or more components in the computing device shown.

[0134] According to another aspect of this disclosure, a computer-readable storage medium is also provided. Figure 9 A schematic diagram of a storage medium 4000 according to the present disclosure is shown.

[0135] like Figure 9As shown, the computer storage medium 4020 stores computer-readable instructions 4010. When the computer-readable instructions 4010 are executed by a processor, the methods according to embodiments of the present disclosure described with reference to the above figures can be performed. The computer-readable storage medium in the embodiments of the present disclosure may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory used in the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0136] This disclosure also provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a method according to an embodiment of this disclosure.

[0137] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0138] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0139] The exemplary embodiments of this disclosure described in detail above are merely illustrative and not restrictive. Those skilled in the art will understand that various modifications and combinations can be made to these embodiments or their features without departing from the principles and spirit of this disclosure, and such modifications should fall within the scope of this disclosure.

Claims

1. A method for processing a training sample set, comprising: Obtain training samples from the first training sample set and determine the representation vector of the training samples; Based on the representation vector of the training sample, a linear combination of concept vectors corresponding to the training sample is determined, wherein the linear combination of concept vectors is a weighted sum of multiple concept vectors that are semantically associated with the training sample; The sampling probability of the training samples is determined based on a linear combination of the concept vectors corresponding to the training samples; and Based on the sampling probability of the training samples, a second training sample set is determined.

2. The method as described in claim 1, wherein, The training samples include sample image data and corresponding text labels for the sample image data. Determining the representation vector of the training samples includes: Based on the sample image data, determine the image representation vector corresponding to the sample image data; Based on the text tag, determine the text representation vector corresponding to the text tag; and Based on the image representation vector corresponding to the sample image data and the text representation vector corresponding to the text label, the representation vector of the training sample is determined; The image representation vector corresponding to the sample image data and the text representation vector corresponding to the text label correspond to the same semantic space.

3. The method as described in claim 2, wherein, The step of determining a linear combination of concept vectors corresponding to the training samples based on the representation vectors of the training samples includes: Obtain a concept dictionary, which includes one or more concept vectors; and Through linear regression optimization, the image representation vector corresponding to the sample image data is transformed into the product of a sparse vector and a concept dictionary. The product of the sparse vector and the concept dictionary is a linear combination of the concept vectors corresponding to the training samples.

4. The method of claim 2, wherein, The step of determining a linear combination of concept vectors corresponding to the training samples based on the representation vectors of the training samples includes: Obtain a concept dictionary, which includes one or more concept vectors; Based on the text representation vectors corresponding to the text labels, determine the concept vectors corresponding to the training samples; and Based on the concept vectors corresponding to the training samples, the image representation vectors corresponding to the sample image data are transformed into the product of a sparse vector and a concept dictionary. The product of the sparse vector and the concept dictionary is a linear combination of the concept vectors corresponding to the training samples.

5. The method as described in claim 3 or 4, wherein, The construction of the concept dictionary includes: Based on each training sample in the first training sample set, determine the text label for each training sample; Extract at least one keyword from the text labels of each training sample; Based on the at least one keyword, determine the concept vector corresponding to the at least one keyword; and The concept dictionary is determined based on the concept vectors corresponding to the at least one keyword.

6. The method of claim 1, wherein, The step of determining the sampling probability of the training samples based on a linear combination of the concept vectors corresponding to the training samples further includes: Obtain the concept dictionary corresponding to the first training sample set, wherein the concept dictionary includes one or more concept vectors; Based on the concept dictionary, a balance index is determined for the first training sample set, the balance index indicating the degree of distribution balance of each concept vector in the concept dictionary among the representation vectors of the training samples in the first training sample set; and The sampling probability of the training samples is determined based on the balance index of the first training sample set.

7. The method of claim 1, wherein, The sampling probability of the training sample indicates the likelihood that the training sample will be included in the second training sample set.

8. The method of claim 6, wherein, Determining the second training sample set includes: Based on the balance index of the first training sample set, at least one key concept is identified; Based on a linear combination of concept vectors corresponding to at least one key concept, augmented training samples are generated; and The expanded training samples are added to the second training sample set.

9. The method of claim 1, further comprising: Based on the second training sample set, train the neural network model. The trained neural network model includes: Obtain the concept dictionary corresponding to the first training sample set, wherein the concept dictionary includes one or more concept vectors; Based on empirical loss and regularization terms, determine the loss function corresponding to the neural network model; and Based on the loss function, the neural network model is trained using the second training sample set; Wherein, the empirical loss indicates the difference between the neural network model's predicted values ​​for training samples in the second training sample set and the true values ​​for training samples in the second training sample set; The regularization term indicates the degree of alignment between the model parameters of the neural network model and at least one concept vector in the concept dictionary.

10. An apparatus for processing a training sample set, comprising: The first module is configured to: acquire training samples from a first training sample set and determine the representation vector of the training samples; The second module is configured to: determine a linear combination of concept vectors corresponding to the training sample based on the representation vector of the training sample, wherein the linear combination of concept vectors is a weighted sum of multiple concept vectors that are semantically associated with the training sample; The third module is configured to: determine the sampling probability of the training samples based on a linear combination of the concept vectors corresponding to the training samples; and The fourth module is configured to determine a second training sample set based on the sampling probability of the training samples.

11. An electronic device, comprising: One or more processors; and One or more memories, wherein the memories store a computer-executable program, which, when executed by the processor, performs the method of any one of claims 1-9.

12. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the method as claimed in any one of claims 1-9.

13. A computer program product comprising computer instructions stored in a computer-readable storage medium, wherein a processor of a computer device reads from the computer-readable medium and executes the computer instructions, causing the computer device to perform the method as described in any one of claims 1-9.

Images