Sample query method, device and storage medium
By selecting representative samples from the unlabeled event sample set in cloud computing scenarios and training the model using pseudo-labels, the problem of high manpower costs was solved, and efficient event threat level assessment and processing priority decision-making were achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ALIBABA CLOUD COMPUTING CO LTD
- Filing Date
- 2023-02-17
- Publication Date
- 2026-07-21
Smart Images
Figure CN116303555B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of machine learning technology, and in particular to a sample query method, device and storage medium. Background Technology
[0002] In cloud computing scenarios, cloud systems generate a massive number of alert messages daily, and multiple alert messages triggered by the same reason are often described as a single event. Faced with a large number of events, users are prone to alert fatigue and cannot judge the threat level of events, thus hindering their ability to prioritize actions. Some solutions prioritize events based on event sequencing models to help users decide the order in which to process alerts.
[0003] Training an event ranking model requires extensive expert experience to analyze and label alarm data samples. However, event labeling is a complex process. For example, an event may include multiple different types of alarms, with varying numbers of different types. Another example is an event containing multiple alarm messages of the same type. The threat level of an event varies depending on the alarm type and the number of alarms. Therefore, labeling event samples requires evaluation and analysis from multiple dimensions, such as alarm type and alarm quantity, resulting in high human resource costs. Thus, a solution is needed. Summary of the Invention
[0004] This application provides a sample query method, device, and storage medium for selecting representative samples from a large number of samples in a sample set for labeling, so as to reduce labor costs for model training.
[0005] This application provides a sample query method, comprising: responding to a sample query request and determining a set of unlabeled event samples; extracting semantic features of multiple event samples in the event sample set; determining a first semantic score for each of the multiple event samples based on the semantic features; the first semantic score of any event sample being used to describe the semantic similarity between the event sample and labeled samples; the labeled samples including event samples in the event sample set labeled with pseudo-labels; selecting a first target event sample from the event sample set based on the first semantic scores of each of the multiple event samples; and the first target event sample being used for labeling to train a machine learning model.
[0006] Optionally, the method further includes: using a clustering algorithm to cluster the multiple event samples to obtain multiple clusters; selecting multiple anchor point samples from the multiple clusters based on the cluster centers of the multiple clusters; and setting pseudo-labels for the multiple anchor point samples to obtain the labeled samples.
[0007] Optionally, determining a first semantic score for each of the plurality of event samples based on their semantic features includes: for a first event sample among the plurality of event samples, determining a first anchor sample in a first cluster and a second anchor sample in a second cluster; the second cluster being any cluster other than the first cluster among the plurality of clusters; calculating a first similarity in semantic features between the first event sample and the first anchor sample, and calculating a second similarity in semantic features between the first event sample and the second anchor sample; and determining the first semantic score for the first event sample based on the first similarity and the second similarity.
[0008] Optionally, after selecting the first target event sample, the method further includes: obtaining the label of the first target event sample; updating the labeled sample using the first target event sample; determining a second semantic score for each of the multiple event samples based on the semantic features of the multiple event samples and the semantic features of the labeled sample; determining a value score for the multiple event samples based on the label of the labeled sample and the semantic features of the multiple event samples; the value score of any event sample is used to describe the difference between the event sample and the labeled sample; selecting a second target event sample from the multiple event samples based on the second semantic score and the value score of the multiple event samples; the second target event sample is used for labeling to optimize the machine learning model.
[0009] Optionally, determining the value score of the plurality of event samples based on the labels of the labeled samples and the semantic features of the plurality of event samples includes: for the second event sample in the event sample set, determining a third event sample and a fourth event sample from the labeled samples whose similarity in semantic features to the second event sample is less than a set similarity threshold; and calculating the value score of the second event sample based on the similarity in semantic features between any two event samples among the semantic features of the second event sample, the third event sample, and the fourth event sample.
[0010] Optionally, the value score of the second event sample is calculated based on the similarity of any pair of event samples in semantic features among the second, third, and fourth event samples. This includes: calculating a third similarity, a fourth similarity, and a fifth similarity between the second and third event samples in semantic features; calculating the difference between the third and fourth similarities, and summing the difference with the fifth similarity; and determining the value score of the second event sample based on the summation result. The value score of the second event sample is negatively correlated with the summation result.
[0011] Optionally, after obtaining the label of the first target event sample, the method further includes: adding the label of the first target event sample to a label set; selecting a second target event sample from the plurality of event samples based on the second semantic score and value score of the plurality of event samples, including: determining a preset adaptive factor when the label set is not empty; performing a weighted calculation on the second semantic score and value score of the plurality of event samples based on the adaptive factor to obtain a fusion score of the plurality of event samples; and selecting a second target event sample from the plurality of event samples based on the fusion score of the plurality of event samples.
[0012] Optionally, the method further includes: sorting the first target event sample according to the label of the first target event sample using an event ranking model to train the event ranking model; and obtaining the label of the second target event sample and sorting the second target event sample according to the label of the second target event sample using the event ranking model to optimize the event ranking model.
[0013] This application also provides an electronic device, including: a memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to perform the steps in the method provided in this application.
[0014] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the method provided in this application.
[0015] In the sample query method provided in this application embodiment, in response to a sample query request, a set of unlabeled event samples can be determined, and semantic features of multiple event samples in the event sample set can be extracted. Based on the semantic features of the multiple event samples and the semantic features of event samples labeled with pseudo-labels, the semantic scores of each of the multiple event samples can be determined, and a target event sample can be selected from the event sample set based on the semantic scores of each of the multiple event samples. In this implementation, active learning can be achieved in a completely unlabeled situation based on pseudo-labels, without the need to obtain the classification labels of the samples in advance; at the same time, by filtering representative event samples in different categories based on the semantic scores of the event samples, the effectiveness of the samples and their representativeness to the overall population can be greatly improved, the number of samples that need to be analyzed and labeled can be reduced, the labor cost can be greatly reduced, and the model training efficiency can be improved. Attached Figure Description
[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0017] Figure 1 A schematic flowchart illustrating a sample query method provided for an exemplary embodiment of this application;
[0018] Figure 2 A schematic diagram of the active learning sample selection process provided for an exemplary embodiment of this application;
[0019] Figure 3 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this application. Detailed Implementation
[0020] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0021] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. “Multiple” generally includes at least two, but does not exclude the inclusion of at least one.
[0022] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.
[0023] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a product or system comprising a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a product or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the product or system that includes said element.
[0024] In sample query scenarios, a typical sample query method is uncertainty sampling. This method extracts samples that are difficult to distinguish from the model and provides them to experts or labelers for annotation, thereby improving algorithm performance more quickly. The key to uncertainty sampling lies in how the samples or data are described. In a least-confidence-based method, samples with lower predicted class probabilities are identified as more "difficult" to distinguish. In a margin sampling method, samples that are very easily classified into two different classes, or samples with small differences in probability of being classified into two different classes, can be selected. In an entropy-based method, entropy is used to measure the uncertainty of a system; higher entropy indicates greater uncertainty, and lower entropy indicates less uncertainty. In binary or multi-class classification scenarios, samples with higher entropy can be selected as data to be labeled. However, the above methods typically use a single model for voting, resulting in selected samples that are uncertain under that model and are not representative.
[0025] Currently, another typical sample query method exists: a committee-based query method. This method selects samples that are "difficult" to distinguish by voting among multiple models. However, this method considers uncertain samples but does not take into account whether the selected samples are class-balanced, thus leading to poor model training performance.
[0026] To address the technical problem that training event ranking models in existing technologies requires extensive expert experience to analyze and label alarm data samples, resulting in high labor costs, this application provides a solution in some embodiments that allows for the retrieval of representative samples from a large amount of sample data, thereby reducing labor costs.
[0027] The technical solutions provided by the various embodiments of this application will be described in detail below with reference to the accompanying drawings.
[0028] Figure 1 This is a flowchart illustrating a sample query method provided in an exemplary embodiment of this application. The method may include, for example: Figure 1 The steps shown are as follows:
[0029] Step 101: Respond to the sample query request and determine the set of unlabeled event samples.
[0030] Step 102: Extract the semantic features of multiple event samples from the event sample set.
[0031] Step 103: Determine the first semantic score of each of the multiple event samples based on their semantic features; the first semantic score of any event sample is used to describe the semantic similarity between the event sample and the labeled samples; the labeled samples include event samples in the event sample set that are labeled with pseudo-labels.
[0032] Step 104: Select a first target event sample from the event sample set based on the first semantic scores of each of the multiple event samples; the first target event sample is used for labeling to train a machine learning model.
[0033] The execution entity in this embodiment can be a terminal device or a server, which runs a sample query model. This sample query model can select high-value event samples from a large number of event samples based on the query request. The terminal device can be a computer, tablet computer, or smartphone. The server can be a conventional server, cloud server, cloud computing data center, elastic computing instance in the cloud, or virtual server in the cloud; this embodiment does not impose any restrictions on this.
[0034] In some scenarios, the event sample set is stored locally on the terminal device. When training an event-related machine learning model, the terminal device can respond to a sample query request, retrieve a portion of representative event samples from the event sample set, and label them, thereby improving model training efficiency. In other scenarios, the event sample set is stored on a server. When training a machine learning model, the terminal device can send a sample query request to the server. The server can respond to the query request, retrieve a portion of representative event samples from its stored event sample set, and provide these samples to the terminal device. The terminal device can then display these representative event samples retrieved by the server for labeling operations.
[0035] The event sample set can include multiple event samples. In this embodiment, an event sample refers to a sample formed by events used for model training. The implementation form of event samples differs in different scenarios. For example, in a business analysis scenario, event samples can be order generation event samples, order cancellation event samples, order after-sales event samples, etc. For example, in a network management scenario, event samples can be network failure event samples, network attack event samples, etc. Furthermore, in a cloud computing scenario, event samples can be event samples composed of alarms generated by the cloud computing platform. Taking a cloud computing scenario as an example, an event sample typically includes multiple alarm messages. For example, an event sample can contain more than 2000 webshell (a malicious script that gains certain privileges on a server through an open port, also known as a web backdoor) type alarms; some event samples can contain 10 alarms, and the types of these 10 alarms can include webshell type alarms, WAF type alarms, and other types of alarms; some event samples can include 2000 multiple WAF (Web Application Firewall) alarms of various types, etc.
[0036] Among them, the event samples in the event sample set are unlabeled samples. In this embodiment, an active learning method can be used to select representative target event samples from the unlabeled event sample set, thereby reducing the number of event samples that need to be labeled while meeting the model training requirements.
[0037] Once the set of event samples is determined, the semantic features of any event sample can be extracted. The semantic features of an event sample refer to the semantic features extracted from the text corresponding to the event sample or the source information of the event sample. These semantic features may include, but are not limited to, at least one of the following: the source features of the event sample, the number of alarms contained in the event sample, and the impact scope of the event sample.
[0038] After determining the semantic features of multiple event samples, a first semantic score can be determined for each of these event samples based on these semantic features. The use of "first" to limit the semantic score is solely for the purpose of distinguishing similar descriptive objects and does not impose any restrictions on the number or order of the described objects.
[0039] The first semantic score of any event sample describes its semantic similarity to the labeled samples. A higher first semantic score indicates greater semantic similarity between the event sample and the labeled samples, and thus a higher probability that the event sample belongs to the same category as the labeled samples. Based on the first semantic score of any event sample, the category to which the event sample belongs, and its representativeness within that category, can be determined.
[0040] In this embodiment, the labeled samples may include event samples labeled with pseudo-labels. The pseudo-labels are temporary labels obtained by approximating the event samples using a sample query model when no actual labels are available. After the subsequent labeling operation of the queried event samples is completed, the pseudo-labels can be replaced with the correctly labeled tags.
[0041] When labeling event samples with pseudo-labels, event samples can be randomly selected from the event sample set, and pseudo-labels can be added to the randomly selected event samples. Alternatively, the event sample set can be clustered, event samples can be selected based on the clustering results, and pseudo-labels can be added to the selected event samples. Specific details will be described in subsequent embodiments and will not be elaborated here.
[0042] After determining the first semantic score for each of the multiple event samples, a first target event sample can be selected from the event sample set. The first target event sample can include one or more event samples. For example, event samples with higher semantic scores than some labeled samples and lower semantic scores than other labeled samples can be selected from the event sample set as representative samples.
[0043] After the sample query model selects the first target event sample, it can use the first target event sample as the query result and return the query result, so that the first target event sample can be used for subsequent labeling and model training.
[0044] In this embodiment, in response to a sample query request, a set of unlabeled event samples can be determined, and semantic features of multiple event samples in the set can be extracted. Based on the semantic features of these multiple event samples and the semantic features of event samples labeled with pseudo-labels, the semantic scores of each of the multiple event samples can be determined, and target event samples can be selected from the set based on their respective semantic scores. In this implementation, active learning can be achieved in a completely unlabeled situation based on pseudo-labels, without the need to obtain the classification labels of the samples in advance. At the same time, by selecting representative event samples from different categories based on the semantic scores of the event samples, the effectiveness of the samples and their representativeness to the overall population can be greatly improved, the number of samples that need to be analyzed and labeled can be reduced, labor costs can be greatly reduced, and the efficiency of model training can be improved.
[0045] In some exemplary embodiments, the labeled samples used to calculate the first semantic score can be obtained through a clustering algorithm. This will be illustrated below.
[0046] Optionally, the sample query model can employ a clustering algorithm to cluster multiple event samples in the event sample set, resulting in multiple clusters. This clustering algorithm may include, but is not limited to: t-SNE (t-distributed stochastic neighbor embedding), K-means clustering, Gaussian Mixture Model (GMM) clustering, or density-based clustering (DBSCAN).
[0047] After obtaining multiple clusters, the sample query model can select multiple anchor samples from these clusters based on their cluster centers. The representative sample selected from any given cluster can be described as the anchor sample within that cluster, such as... Figure 2 The anchor samples shown are represented by light-colored dots. Within a cluster, one or more event samples located at or near the cluster center can be selected as anchor samples for that cluster. After determining the anchor samples for each of the multiple clusters, the sample query model can assign pseudo-labels to these anchor samples and treat the anchor samples with pseudo-labels as labeled samples. For example... Figure 2As shown, after obtaining the anchor point samples, the semantic score calculation can proceed based on the anchor point samples. Correspondingly, when the sample query model determines the first semantic score of each of the multiple event samples based on their semantic features, it can determine the first semantic score of each of the multiple event samples based on both the semantic features of the multiple event samples and the semantic features of the multiple anchor point samples. The following explanation will use the first event sample as an example; this first event sample can be any event sample.
[0048] Optionally, for a first event sample among the multiple event samples, the sample query model can determine the anchor samples in the cluster where the first event sample belongs, as well as the anchor samples in other clusters besides the cluster where the first event sample belongs. For ease of description and distinction, the cluster where the first event sample belongs is described as the first cluster, and the anchor samples in the first cluster are described as the first anchor samples; the other clusters besides the first cluster are described as the second cluster, and the anchor samples in the second cluster are described as the second anchor samples. The second cluster may include one or more clusters, the first anchor sample may include one or more anchor samples, and the second anchor sample may include one or more anchor samples.
[0049] After determining the first anchor sample and the second anchor sample, the sample query model can calculate the first similarity in semantic features between the first event sample and the first anchor sample, and the second similarity in semantic features between the first event sample and the second anchor sample. A higher first similarity indicates a greater probability that the first event sample belongs to the category of the first anchor sample. A lower second similarity indicates a lower probability that the first event sample belongs to the category of the second anchor sample. That is, if the first similarity of the first event sample is high and the second similarity is low, the first event sample can be considered a relatively representative sample in the first cluster.
[0050] The sample query model can determine the first semantic score of the first event sample based on the first similarity and the second similarity of the first event sample. For example, the first semantic score of the first event sample can be the larger of the first similarity and the second similarity, the ratio of the first similarity to the second similarity, or the difference between the first similarity and the second similarity. This embodiment does not impose any restrictions.
[0051] For example, in some embodiments, the semantic score can be calculated using the following formula:
[0052]
[0053] Where S1 represents the semantic score, σ[.] is the "min-max" normalization function, and Z sThe semantic features of the event sample. This represents the j-th unlabeled event sample in the first event sample set. This represents the labeled sample corresponding to the j-th event sample, where j = 0, 1, 2…N, and N is the total number of samples in the first event sample set.
[0054] In this implementation, anchor samples in the first cluster containing the first event sample can be considered as positive examples of the first event sample, and anchor samples in the second cluster can be considered as negative examples of the first event sample. By calculating the semantic scores of the first event sample, positive examples, and negative examples, contrastive learning can be used to semantically distinguish event samples of the same category from event samples of different categories, thereby learning the invariant semantic attributes of event samples.
[0055] In the above embodiments, the first target event sample selected by the sample query model is a representative event sample of different categories, thereby meeting the requirement for sample diversity in the process of training the machine learning model. During the training process of the machine learning model, multiple rounds of training can be performed to further optimize the model. In addition to using diverse samples to train the machine learning model during these multiple rounds of training, discriminative samples can also be used to further train the model, thereby improving its ability to distinguish samples that are difficult to differentiate. That is, the sample query model can perform multiple rounds of sample selection operations. The first round of sample selection operations can be referred to the description in the foregoing embodiments. The second round of sample selection operations will be described exemplarily below.
[0056] In some exemplary embodiments, after the sample query model selects a first target event sample, it can output the first target event sample through a terminal device, so that users can label the first target event sample based on their professional experience. For example, in a cloud computing scenario, a danger level label can be added to the selected event sample. For example, in a network management scenario, labels such as network failure cause label and network attack type label can be added to the selected event sample.
[0057] Optionally, during the second round of sample screening, the sample query model can obtain the label of the first target event sample and use the first target event sample to update the labeled sample. That is, multiple anchor point samples in the labeled sample can be replaced by the first target event sample. Figure 2As shown in the "Update" step, after the semantic score calculation is completed, the fusion score calculation step begins. In the first round of sample selection, the fusion score is the semantic score. After selecting target event samples based on this fusion score, the anchor samples are updated using the target event samples. This allows the pseudo-labels required for active learning to be replaced with real labels. After updating the labeled samples, the sample query model can determine the second semantic score of each of the multiple event samples in the event sample set, based on the semantic features of the labeled samples and the semantic features of the event samples themselves. Specifically, when determining the second semantic score of any event sample, the similarity (i.e., semantic similarity) between the first event sample and the first target event sample in terms of semantic features can be calculated, and the second semantic score of the event sample can be determined based on this similarity. For details, please refer to the description in the foregoing embodiments; further elaboration is not provided here.
[0058] In the second round of sample selection, in addition to calculating the semantic score of the event samples, a value score can also be calculated. For example... Figure 2 As shown, the updated anchor samples are input into the semantic score calculation stage and the value score calculation stage to calculate the semantic score and value score. The value score of any event sample is used to describe the difference between the event sample and the labeled sample, thereby selecting event samples with higher learning value that are different from those in the first round during the second round of sample selection.
[0059] The sample query model can determine the value score of multiple event samples based on the labels of the labeled samples and the semantic features of the multiple event samples. The following will use the second event sample as an example for illustration, where the second event sample can be any event sample.
[0060] Optionally, for the second event sample in the event sample set, the sample query model can determine, from the labeled samples, a third event sample and a fourth event sample whose semantic similarity to the second event sample is less than a set similarity threshold. The second event sample has a high probability of belonging to the same category as the third and fourth event samples.
[0061] The semantic similarity between different event samples can be represented by clustering distance. In some embodiments, for an unlabeled second event sample, a third and fourth event sample with a true label can be searched from the labeled samples based on their clustering radius, provided they are close to the second event sample in terms of clustering distance. The labels of the third and fourth event samples can be the same or different; this embodiment does not impose any restrictions.
[0062] The sample query model can calculate the value score of the second event sample based on the similarity of semantic features between any two event samples among the second, third, and fourth event samples.
[0063] Optionally, when calculating the value score of the second event sample, the sample query model may calculate a third similarity in semantic features between the second event sample and the third event sample, a fourth similarity in semantic features between the second event sample and the fourth event sample, and a fifth similarity in semantic features between the third event sample and the fourth event sample. After determining the above similarities, the sample query model may calculate the value score of the second event sample based on the difference between the third and fourth similarities, and the sum of the difference and the fifth similarity.
[0064] The second and third similarity scores measure the similarity between the second event sample and the labeled third and fourth event samples. A larger difference between the third and fourth similarity scores indicates a higher similarity between the second event sample and the labeled event samples. The fourth similarity score reflects the information content of the unlabeled samples; a higher fourth similarity score indicates a lower information content of the unlabeled samples.
[0065] Optionally, the value score can be negatively correlated with the sum of similarities. That is, the larger the sum of similarities, the greater the similarity between the second event sample and the labeled event samples, and the lower the information content of the unlabeled event samples, resulting in a lower value score for the second event sample. Conversely, the smaller the sum of similarities, the smaller the semantic similarity between the second event sample and the labeled event samples, and the greater the information content of the unlabeled event samples, resulting in a higher value score for the second event sample. A higher value score indicates higher sample discriminability and information content, which is more conducive to model optimization.
[0066] For example, in some embodiments, the value score can be calculated using the following formula:
[0067]
[0068] Where S2 represents the value score, σ[.] is the "min-max" normalization function, and Z d The semantic features of the event sample. This represents the i-th unlabeled event sample in the first event sample set. Indicates and The clustering distance is relatively close, and it is an event sample with a true label (i.e., the third event sample). Indicates and Another event sample (i.e., the fourth event sample) whose cluster distance is relatively close and which has a true label. i = 0, 1, 2...N, where N is the total number of samples in the first event sample set.
[0069] Based on the above steps, the second semantic score and value score of each of the multiple event samples in the event sample set can be calculated. The sample query model can then select a second target event sample from these multiple event samples based on their second semantic scores and value scores.
[0070] In some optional embodiments, event samples with a second semantic score greater than a first score threshold and a second value score greater than a second score threshold can be selected from the plurality of event samples as the second target event samples.
[0071] In some alternative embodiments, a weighted score of a second semantic score and a value score can be calculated for any event sample. Event samples with a weighted score greater than a third score threshold are selected as the second target event samples. This embodiment does not impose any restrictions. In this implementation, when filtering event samples, both the semantic score and the value score of the event sample are considered simultaneously. The semantic score can reduce the risk of querying a large number of redundant samples in the class distribution, and the value score can reduce the risk of invalid queries, further improving sample quality.
[0072] In some exemplary embodiments, when performing multi-round sample selection operations, the sample query model can automatically adapt to different rounds of sample query scenarios by controlling the weighting factor between semantic score and value score. The weighting factor for the value score can be described as an adaptive factor. The value of this adaptive factor is related to whether a real label exists in the label set. When the label set is empty (i.e., no samples with real labels have been obtained), the adaptive factor can be set to 0. In this case, the sample query model can prioritize selecting representative samples from each category, i.e., the first target event sample, based on the semantic score. After selecting the first target event sample, the label of the first target event sample can be obtained and added to the label set.
[0073] like Figure 2 As shown, the sample query model contains a label set, and the model can determine whether the label set is empty in each round of filtering. When the label set is not empty, the adaptive factor can be set to a non-zero value. For example, the adaptive factor can be 1, 0.8, 0.7, 0.5, or any non-zero number. The specific choice can be made according to actual needs, and this embodiment does not impose any restrictions.
[0074] Accordingly, when the sample query model selects a second target event sample from multiple event samples based on their second semantic scores and value scores, it can determine a preset adaptive factor if the label set is not empty. Based on this adaptive factor, it then performs a weighted calculation of the second semantic scores and value scores of the multiple event samples to obtain a fusion score for the multiple event samples. That is, in the second round of sample selection, Figure 2 The fusion score shown can be obtained by weighted fusion of semantic score and value score. The sample query model selects a second target event sample from the multiple event samples based on the fusion score of these multiple event samples.
[0075] In this implementation, an adaptive factor can be set based on whether the label set is empty, thus adapting to different sample selection scenarios based on the adaptive factor event. By controlling the adaptive factor, diverse samples can be selected in the first round of sample selection to ensure that the machine learning model can distinguish most samples. By controlling the adaptive factor, samples with greater differences from the labeled samples can be selected in the second round of sample selection to ensure that the machine learning model can learn to distinguish more difficult samples and reduce misclassification.
[0076] After selecting target event samples based on the above embodiments, the machine learning model can be trained using these samples. This machine learning model can be an event classification model, an event recommendation model, an event prediction model, or an event ranking model, etc., and the specific model can be determined according to the application scenario. This embodiment does not impose any limitations on this. The following will use an event ranking model as an example for illustrative explanation.
[0077] After obtaining the target event samples, the event ranking model can be trained. The event ranking model can be implemented based on a gated recurrent unit (GRU), a recurrent neural network (RNN), a long short-term memory (LSTM) network, or a variation of the above models. This embodiment does not impose any restrictions on this.
[0078] In some optional embodiments, the event ranking model can be trained in batches. Specifically, in the first training batch, a first target event sample can be selected from the event sample set based on its semantic score; and the first target event sample is then ranked using the event ranking model based on its label, thereby training the event ranking model. The first target event sample is a representative sample in the event sample set, allowing the event ranking model to be trained to recognize diverse samples. If the event ranking model converges to a specified state (e.g., the loss function is less than a certain threshold), samples can be selected again, and a second batch of training can be performed. In the second training batch, a second target event sample can be selected from the event sample set based on its semantic and value scores; and the second target event sample is then ranked using the event ranking model based on its label, thereby optimizing the event ranking model. The second target event sample is a more difficult sample to distinguish, allowing the trained event ranking model to further learn its ability to distinguish more difficult samples, thus improving ranking performance.
[0079] In some alternative embodiments, the event ranking model can be trained centrally after selecting the first target event samples and the second event samples in batches. In this implementation, the event ranking model can be used to rank the first target event samples according to their labels to train the event ranking model. After the event ranking model has initially converged, the event ranking model can be used to rank the second target event samples according to their labels to optimize the event ranking model, which will not be elaborated further.
[0080] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 to 104 can be device A; or the execution subject of steps 101 and 102 can be device A, and the execution subject of step 103 can be device B; and so on.
[0081] Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel.
[0082] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.
[0083] It should be noted that the terms "first" and "second" in this article are used to distinguish different messages, devices, modules, etc., and do not represent a chronological order, nor do they limit "first" and "second" to different types.
[0084] Figure 3 This illustration shows a structural diagram of an electronic device provided in an exemplary embodiment of this application, which is applicable to the sample query method provided in the foregoing embodiments. For example... Figure 3 As shown, the electronic device includes a memory 301 and a processor 302.
[0085] Memory 301 is used to store computer programs and can be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device.
[0086] The memory 301 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.
[0087] Processor 302, coupled to memory 301, is configured to execute a computer program in memory 301 for: responding to a sample query request and determining a set of unlabeled event samples; extracting semantic features of multiple event samples from the event sample set; determining a first semantic score for each of the multiple event samples based on the semantic features; the first semantic score of any event sample is used to describe the semantic similarity between the event sample and labeled samples; the labeled samples include: event samples in the event sample set labeled with pseudo-labels; selecting a first target event sample from the event sample set based on the first semantic scores of each of the multiple event samples; the first target event sample is used for labeling to train a machine learning model.
[0088] Optionally, the processor 302 is further configured to: employ a clustering algorithm to cluster the multiple event samples to obtain multiple clusters; select multiple anchor point samples from the multiple clusters based on the cluster centers of the multiple clusters; and set pseudo labels for the multiple anchor point samples to obtain the labeled samples.
[0089] Optionally, when determining the first semantic score of each of the plurality of event samples based on the semantic features of the plurality of event samples, the processor 302 specifically performs the following steps: for the first event sample among the plurality of event samples, determine a first anchor sample in a first cluster and a second anchor sample in a second cluster; the second cluster is a cluster other than the first cluster among the plurality of clusters; calculate a first similarity in semantic features between the first event sample and the first anchor sample, and calculate a second similarity in semantic features between the first event sample and the second anchor sample; and determine the first semantic score of the first event sample based on the first similarity and the second similarity.
[0090] Optionally, after selecting the first target event sample, the processor 302 is further configured to: obtain the label of the first target event sample; update the labeled sample using the first target event sample; determine the second semantic score of each of the multiple event samples based on the semantic features of the multiple event samples and the semantic features of the labeled sample; determine the value score of the multiple event samples based on the label of the labeled sample and the semantic features of the multiple event samples; the value score of any event sample is used to describe the difference between the event sample and the labeled sample; select a second target event sample from the multiple event samples based on the second semantic score and the value score of the multiple event samples; the second target event sample is used for labeling to optimize the machine learning model.
[0091] Optionally, when determining the value scores of the plurality of event samples based on the labels of the labeled samples and the semantic features of the plurality of event samples, the processor 302 specifically performs the following: for the second event sample in the event sample set, from the labeled samples, determine a third event sample and a fourth event sample whose similarity in semantic features to the second event sample is less than a set similarity threshold; calculate the value score of the second event sample based on the similarity in semantic features between any two event samples among the semantic features of the second event sample, the third event sample, and the fourth event sample.
[0092] Optionally, when the processor 302 calculates the value score of the second event sample based on the semantic similarity of any pair of event samples among the semantic features of the second event sample, the third event sample, and the fourth event sample, it specifically performs the following steps: calculating a third similarity between the second event sample and the third event sample in terms of semantic features; calculating a fourth similarity between the second event sample and the fourth event sample in terms of semantic features; calculating a fifth similarity between the third event sample and the fourth event sample in terms of semantic features; calculating the difference between the third similarity and the fourth similarity, and summing the difference with the fifth similarity; determining the value score of the second event sample based on the result of the summation calculation; the value score of the second event sample is negatively correlated with the result of the summation calculation.
[0093] Optionally, after acquiring the label of the first target event sample, the processor 302 is further configured to: add the label of the first target event sample to a label set; and select a second target event sample from the plurality of event samples based on the second semantic score and value score of the plurality of event samples, including: determining a preset adaptive factor when the label set is not empty; performing a weighted calculation on the second semantic score and value score of the plurality of event samples based on the adaptive factor to obtain a fusion score of the plurality of event samples; and selecting a second target event sample from the plurality of event samples based on the fusion score of the plurality of event samples.
[0094] Optionally, the processor 302 is further configured to: sort the first target event sample according to the label of the first target event sample using an event sorting model to train the event sorting model; and obtain the label of the second target event sample, and sort the second target event sample according to the label of the second target event sample using the event sorting model to optimize the event sorting model.
[0095] Furthermore, such as Figure 3 As shown, the electronic device also includes other components such as a communication component 303, a power supply component 304, a display component 305, and an audio component 306. Figure 3 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 3 The components shown.
[0096] The communication component 303 is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as Wi-Fi, 2G (e.g., Global System for Mobile Communications (GSM)), 3G (e.g., Wideband Code Division Multiple Access (WCDMA), 4G (e.g., Long Term Evolution (LTE)), 4G+ (e.g., LTE-Advanced (LTE-A)), or 5G (5th Generation Mobile Communication Technology), or combinations thereof. In an exemplary embodiment, the communication component may be implemented based on Near Field Communication (NFC), Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), Bluetooth (BT), and other technologies.
[0097] The power supply component 304 is used to provide power to various components of the device in which the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component is located.
[0098] The display component 305 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation.
[0099] The audio component 306 can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as a call mode, recording mode, or voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.
[0100] In this embodiment, in response to a sample query request, the electronic device can determine a set of unlabeled event samples and extract semantic features from multiple event samples in the set. Based on the semantic features of these multiple event samples and the semantic features of event samples labeled with pseudo-labels, the semantic scores of each of the multiple event samples can be determined. Based on these semantic scores, a target event sample is selected from the set of event samples. In this implementation, active learning can be achieved in a completely unlabeled situation based on pseudo-labels, without the need to pre-obtain classification labels for the samples. Simultaneously, by filtering representative event samples from different categories based on their semantic scores, the effectiveness and representativeness of the samples to the overall population can be greatly improved. This reduces the number of samples that need to be analyzed and labeled, significantly lowers labor costs, and improves model training efficiency.
[0101] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be performed by an electronic device in the above method embodiments.
[0102] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0103] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0104] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0105] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0106] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0107] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0108] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0109] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0110] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.
Claims
1. A sample query method, characterized in that, include: Respond to sample query requests and determine the set of unlabeled event samples; Extract semantic features from multiple event samples in the event sample set; In the first round of sample screening, the first semantic score of each of the multiple event samples is determined based on their semantic features. The first semantic score of any event sample is used to describe the semantic similarity between the event sample and the labeled samples; The labeled samples include: event samples in the event sample set that are labeled with pseudo-labels; Based on the first semantic scores of each of the plurality of event samples, a first target event sample is selected from the event sample set; the first target event sample is used for labeling to train a machine learning model. In the second round of sample screening, event samples that are different from those in the first round are selected from the multiple event samples based on their value scores. The value score of any event sample is used to describe the difference between the event sample and the updated labeled sample. The updated labeled sample is updated based on the first target event sample after labeling.
2. The method according to claim 1, characterized in that, Also includes: A clustering algorithm is used to cluster the multiple event samples to obtain multiple clusters; Based on the cluster centers of the multiple clusters, select multiple anchor point samples from the multiple clusters; Pseudo-labels are set for the multiple anchor point samples to obtain the labeled samples.
3. The method according to claim 2, characterized in that, Based on the semantic features of the multiple event samples, a first semantic score is determined for each of the multiple event samples, including: For a first event sample among the plurality of event samples, determine a first anchor sample in a first cluster and a second anchor sample in a second cluster; the second cluster is a cluster other than the first cluster among the plurality of clusters; Calculate the first similarity in semantic features between the first event sample and the first anchor sample, and calculate the second similarity in semantic features between the first event sample and the second anchor sample; The first semantic score of the first event sample is determined based on the first similarity and the second similarity.
4. The method according to claim 1, characterized in that, Based on the value scores of the multiple event samples, event samples that differ from those in the first round are selected from the multiple event samples, including: Obtain the label of the first target event sample; The labeled samples are updated using the first target event sample; Based on the semantic features of the multiple event samples and the semantic features of the labeled samples, a second semantic score is determined for each of the multiple event samples; Based on the labels of the labeled samples and the semantic features of the multiple event samples, the value scores of the multiple event samples are determined. Based on the second semantic score and value score of the plurality of event samples, a second target event sample is selected from the plurality of event samples; the second target event sample is used for labeling to optimize the machine learning model.
5. The method according to claim 4, characterized in that, Based on the labels of the labeled samples and the semantic features of the multiple event samples, the value scores of the multiple event samples are determined, including: For the second event sample in the event sample set, a third event sample and a fourth event sample are determined from the labeled samples that have a semantic similarity of less than a set similarity threshold with the second event sample; The value score of the second event sample is calculated based on the similarity of any pair of event samples in semantic features among the second event sample, the third event sample, and the fourth event sample.
6. The method according to claim 5, characterized in that, The value score of the second event sample is calculated based on the similarity of semantic features between any two event samples from the second, third, and fourth event samples, including: Calculate the third similarity of the second event sample and the third event sample in terms of semantic features, the fourth similarity of the second event sample and the fourth event sample in terms of semantic features, and the fifth similarity of the third event sample and the fourth event sample in terms of semantic features; Calculate the difference between the third similarity and the fourth similarity, and sum the difference with the fifth similarity. Based on the summation result, the value score of the second event sample is determined; the value score of the second event sample is negatively correlated with the summation result.
7. The method according to claim 4, characterized in that, After obtaining the label of the first target event sample, the method further includes: Add the label of the first target event sample to the label set; Based on the second semantic score and value score of the plurality of event samples, a second target event sample is selected from the plurality of event samples, including: When the tag set is not empty, a preset adaptive factor is determined; Based on the adaptive factor, the second semantic score and value score of the multiple event samples are weighted and calculated to obtain the fusion score of the multiple event samples; Based on the fusion scores of the multiple event samples, a second target event sample is selected from the multiple event samples.
8. The method according to any one of claims 4-7, characterized in that, Also includes: Based on the labels of the first target event samples, an event ranking model is used to rank the first target event samples in order to train the event ranking model. as well as, The labels of the second target event samples are obtained, and the event ranking model is used to rank the second target event samples according to the labels of the second target event samples in order to optimize the event ranking model.
9. An electronic device, characterized in that, include: Memory and processor; The memory is used to store one or more computer instructions; The processor is configured to execute one or more computer instructions for performing the steps of the method according to any one of claims 1-8.
10. A computer-readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it is able to perform the steps of the method described in any one of claims 1-8.