Hard example mining method and device, electronic equipment and storage medium

Abstract

The application provides a difficult example mining method and device, electronic equipment and storage medium. Including: obtaining a basic positioning model and a test data set corresponding to the basic positioning model; grouping a plurality of test data using the basic positioning model to obtain a plurality of first data groups; for each first data group, determining good example data and difficult example data from a plurality of test data in the first data group based on audio data in the first data group; optimizing the basic positioning model using the good example data to obtain an updated positioning model; grouping a plurality of difficult example data using the updated positioning model to obtain a plurality of second data groups; for each second data group, determining target difficult example data from a plurality of difficult example data in the second data group based on audio data and video data in the second data group. Thus, intelligent mining of difficult example data is achieved.

Description

Technical Field

[0001] This application relates to the field of data mining technology, and in particular to a method, apparatus, electronic device and storage medium for mining difficult cases. Background Technology

[0002] With the development of artificial intelligence technology, speaker localization models can be pre-trained and used to locate the speakers corresponding to lines in film and television works. In applications, speaker localization models can generally locate the corresponding speakers based on audio and video data corresponding to the lines.

[0003] In practical applications, speaker localization models achieve relatively accurate localization results for certain audio and video data, which can be referred to as goodcase data. Conversely, the same speaker localization model may not achieve accurate localization results for other audio and video data, which can be referred to as badcase data. If these badcase data can be labeled and used for iterative training of the speaker localization model, the localization performance of the model can be greatly improved.

[0004] However, how to identify difficult case data corresponding to the speaker localization model from a large amount of unlabeled audio and video data is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, electronic device, and storage medium for mining difficult examples, in order to solve the current problem of the difficulty in mining difficult example data. The specific technical solution is as follows:

[0006] Firstly, this application provides a method for hard example mining, including:

[0007] Obtain a basic localization model and a test dataset corresponding to the basic localization model, wherein the basic localization model is used to locate the speaker, and the test dataset contains multiple test data, each of which contains audio data and video data;

[0008] The basic localization model is used to group multiple test data to obtain multiple first data groups, wherein each first data group contains test data corresponding to the same speaker identified by the basic localization model;

[0009] For each first data group, good example data and difficult example data are determined from multiple test data in the first data group based on the audio data in the first data group;

[0010] The base positioning model is optimized using the good case data to obtain an updated positioning model;

[0011] The updated localization model is used to group multiple difficult case data to obtain multiple second data groups, wherein each second data group contains difficult case data corresponding to the same speaker identified by the updated localization model;

[0012] For each second data group, based on the audio and video data in the second data group, target difficult data is determined from multiple difficult data in the second data group.

[0013] In one possible implementation, determining good and difficult test data from multiple test data sets in the first data group based on audio data in the first data group includes:

[0014] For each audio data in the first data group, the similarity between the audio data and each other audio data is calculated to obtain at least one first similarity value;

[0015] Calculate the average value corresponding to at least one of the first similarity values, and use the average value as the first similarity score corresponding to the audio data;

[0016] Audio data with a first similarity score greater than or equal to a first preset threshold are identified as good example audio, and audio data with a first similarity score less than the first preset threshold are identified as difficult example audio.

[0017] The test data corresponding to the good example audio is determined as good example data, and the test data corresponding to the difficult example audio is determined as difficult example data.

[0018] In one possible implementation, determining the target difficult case data from multiple difficult case data in the second data group based on the audio and video data in the second data group includes:

[0019] For each difficult example data in the second data group, the similarity between the audio data in the difficult example data and each other audio data is calculated to obtain at least one second similarity value; and the similarity between the video data in the difficult example data and each other video data is calculated to obtain at least one third similarity value.

[0020] Calculate the average value corresponding to at least one of the second similarity values, and use the average value as the second similarity score corresponding to the difficult example data; and calculate the average value corresponding to at least one of the third similarity values, and use the average value as the third similarity score corresponding to the difficult example data.

[0021] Based on the second and third similarity scores corresponding to each difficult example data, the target difficult example data is determined from multiple difficult example data in the second data group.

[0022] In one possible implementation, determining the target difficult example data from multiple difficult example data in the second data group based on the second similarity score and the third similarity score corresponding to each difficult example data includes:

[0023] The second and third similarity scores corresponding to each difficult example data are input into the support vector machine, so that the support vector machine can filter out the target difficult example data from multiple difficult example data based on the second and third similarity scores corresponding to each difficult example data.

[0024] In one possible implementation, determining the target difficult example data from multiple difficult example data in the second data group based on the second similarity score and the third similarity score corresponding to each difficult example data includes:

[0025] For each difficult example data, the second similarity score and the third similarity score corresponding to the difficult example data are weighted and summed to obtain the target similarity score;

[0026] Difficult example data whose corresponding target similarity score is less than the second preset threshold is identified as the target difficult example data.

[0027] In one possible implementation, after determining the target difficult case data from multiple difficult case data in the second data group based on the audio and video data in the second data group, the method further includes:

[0028] The difficult case data in the second data group, excluding the target difficult case data, are identified as the target good case data;

[0029] The updated localization model is optimized using the target good case data to obtain the target localization model.

[0030] In one possible implementation, after determining the target difficult case data from multiple difficult case data in the second data group based on the audio and video data in the second data group, the method further includes:

[0031] The target difficult example data is displayed through visualization components;

[0032] Receive instruction information input by the user based on the target difficulty example data;

[0033] If the indication information indicates that the target difficult example data is noisy data, the noisy data is labeled.

[0034] Secondly, this application provides a difficult-to-excavate device, comprising:

[0035] The acquisition module is used to acquire a basic positioning model and a test dataset corresponding to the basic positioning model. The basic positioning model is used to locate the speaker, and the test dataset contains multiple test data, each of which contains audio data and video data.

[0036] The first grouping module is used to group multiple test data using the basic positioning model to obtain multiple first data groups, wherein each first data group contains test data corresponding to the same speaker identified by the basic positioning model;

[0037] The first determining module is used to determine good sample data and difficult sample data from multiple test data in the first data group based on the audio data in the first data group for each first data group;

[0038] The optimization module is used to optimize the basic positioning model using the good case data to obtain an updated positioning model;

[0039] The second grouping module is used to group multiple difficult case data using the updated localization model to obtain multiple second data groups, wherein each second data group contains difficult case data corresponding to the same speaker identified by the updated localization model;

[0040] The second determining module is used to determine target difficult data from multiple difficult data in the second data group based on the audio data and video data in the second data group for each second data group.

[0041] In one possible implementation, the first determining module is further configured to:

[0042] For each audio data in the first data group, the similarity between the audio data and each other audio data is calculated to obtain at least one first similarity value;

[0043] Calculate the average value corresponding to at least one of the first similarity values, and use the average value as the first similarity score corresponding to the audio data;

[0044] Audio data with a first similarity score greater than or equal to a first preset threshold are identified as good example audio, and audio data with a first similarity score less than the first preset threshold are identified as difficult example audio.

[0045] The test data corresponding to the good example audio is determined as good example data, and the test data corresponding to the difficult example audio is determined as difficult example data.

[0046] In one possible implementation, the second determining module is further configured to:

[0047] For each difficult example data in the second data group, the similarity between the audio data in the difficult example data and each other audio data is calculated to obtain at least one second similarity value; and the similarity between the video data in the difficult example data and each other video data is calculated to obtain at least one third similarity value.

[0048] Calculate the average value corresponding to at least one of the second similarity values, and use the average value as the second similarity score corresponding to the difficult example data; and calculate the average value corresponding to at least one of the third similarity values, and use the average value as the third similarity score corresponding to the difficult example data.

[0049] Based on the second and third similarity scores corresponding to each difficult example data, the target difficult example data is determined from multiple difficult example data in the second data group.

[0050] In one possible implementation, the second determining module is further configured to:

[0051] The second and third similarity scores corresponding to each difficult example data are input into the support vector machine, so that the support vector machine can filter out the target difficult example data from multiple difficult example data based on the second and third similarity scores corresponding to each difficult example data.

[0052] In one possible implementation, the second determining module is further configured to:

[0053] For each difficult example data, the second similarity score and the third similarity score corresponding to the difficult example data are weighted and summed to obtain the target similarity score;

[0054] Difficult example data whose corresponding target similarity score is less than the second preset threshold is identified as the target difficult example data.

[0055] In one possible implementation, the optimization module is further configured to:

[0056] The difficult case data in the second data group, excluding the target difficult case data, are identified as the target good case data;

[0057] The updated localization model is optimized using the target good case data to obtain the target localization model.

[0058] In one possible implementation, the device further includes a labeling module for:

[0059] The target difficult example data is displayed through visualization components;

[0060] Receive instruction information input by the user based on the target difficulty example data;

[0061] If the indication information indicates that the target difficult example data is noisy data, the noisy data is labeled.

[0062] Thirdly, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0063] Memory, used to store computer programs;

[0064] When a processor executes a program stored in memory, it implements any of the steps described in the first aspect.

[0065] Fourthly, a computer-readable storage medium is provided, characterized in that the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any of the methods described in the first aspect.

[0066] Fifthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to execute any of the aforementioned methods for mining difficult cases.

[0067] Beneficial effects of the embodiments in this application:

[0068] This application provides a method, apparatus, electronic device, and storage medium for difficult example mining. In this embodiment, firstly, a basic localization model and its corresponding test dataset are obtained. Then, multiple test data points in the test dataset are grouped using the basic localization model to obtain multiple first data groups. Next, for each first data group, good example data and difficult example data are determined from the multiple test data points within that first data group based on the audio data in that first data group. The basic localization model is then optimized using the good example data to obtain an updated localization model. Subsequently, the multiple difficult example data points are grouped using the updated localization model to obtain multiple second data groups. Finally, for each second data group, target difficult example data is determined from the multiple difficult example data points within that second data group based on the audio and video data in that second data group. This achieves intelligent mining of difficult example data, solving the current problem of the difficulty in mining difficult example data.

[0069] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0070] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0071] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0072] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0073] Figure 1 A flowchart illustrating a difficult instance mining method provided in this application embodiment;

[0074] Figure 2 A flowchart illustrating another hard case mining method provided in this application embodiment;

[0075] Figure 3 This is a schematic diagram of the structure of a difficult excavation device provided in an embodiment of this application;

[0076] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0078] The following disclosure provides numerous different embodiments or examples for implementing various structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of the invention. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0079] Figure 1This is a flowchart illustrating a hard case mining method provided in an embodiment of this application. This method can be applied to one or more electronic devices such as smartphones, laptops, desktop computers, portable computers, and servers. Furthermore, the execution entity of this method can be hardware or software. When the execution entity is hardware, it can be one or more of the aforementioned electronic devices. For example, a single electronic device can execute this method, or multiple electronic devices can cooperate with each other to execute this method. When the execution entity is software, this method can be implemented as multiple software programs or software modules, or as a single software program or software module. No specific limitations are made here.

[0080] like Figure 1 As shown, the method specifically includes:

[0081] S101, Obtain the basic positioning model and the test dataset corresponding to the basic positioning model, wherein the basic positioning model is used to locate the speaker, and the test dataset contains multiple test data, each of which contains audio data and video data.

[0082] The basic localization model is used to locate the corresponding speaker based on the audio and video data corresponding to the dialogue. The audio data refers to the audio portion between the start and end times of the dialogue, and the video data refers to the video portion between the start and end times of the dialogue.

[0083] A test dataset refers to a collection of multiple test data sets, each containing corresponding audio and video data.

[0084] In this embodiment of the application, a user-uploaded or specified model can be used as the basic localization model, and a user-uploaded or specified dataset can be used as the test dataset.

[0085] S102, the basic positioning model is used to group the multiple test data to obtain multiple first data groups, wherein each first data group contains test data corresponding to the same speaker identified by the basic positioning model.

[0086] In this embodiment of the application, the audio features of the audio data and the video features of the video data in each test data can be extracted using the basic localization model. Based on the audio features and video features, the test data of the same speaker can be located. Then, the test data corresponding to the same speaker can be divided into the same first data group.

[0087] S103, for each first data group, determine good example data and difficult example data from multiple test data in the first data group based on the audio data in the first data group.

[0088] In this embodiment of the application, the specific implementation of determining good sample data and difficult sample data from multiple test data in the first data group based on the audio data in the first data group may include the following steps:

[0089] Step A1: For each audio data in the first data group, calculate the similarity between the audio data and each other audio data to obtain at least one first similarity value;

[0090] Step A2: Calculate the average value corresponding to at least one of the first similarity values, and use the average value as the first similarity score corresponding to the audio data;

[0091] Step A3: Audio data with a first similarity score greater than or equal to a first preset threshold are identified as good example audio, and audio data with a first similarity score less than the first preset threshold are identified as difficult example audio.

[0092] Step A4: The test data corresponding to the good example audio is determined as good example data, and the test data corresponding to the difficult example audio is determined as difficult example data.

[0093] In this scheme, for each first data group, firstly, based on the audio features of each audio data in the data group, the first similarity score corresponding to each audio data is calculated. Specifically, for each audio data, the cosine similarity value (i.e., the first similarity value) between the audio features of the audio data and the audio features of each other audio data is determined. Then, all the cosine similarity values ​​corresponding to the audio data are summed and averaged to obtain its corresponding average cosine similarity xi (i = 1, 2, 3... n) (i.e., the first similarity score).

[0094] Finally, by setting a first preset threshold Audio_T, data with excessive differences (i.e., audio data with similarity lower than Audio_T) are filtered out as hard data, and the test data corresponding to the hard data are determined as hard data. The other audio is good data, and the test data corresponding to the good data are determined as good data, which are used for subsequent model training.

[0095] S104, optimize the basic positioning model using the good case data to obtain an updated positioning model.

[0096] In this embodiment, the obtained good example data is expanded into the training set of the basic positioning model, and the basic positioning model is retrained using the expanded training set to complete the fine-tuning of the new model (i.e., the updated positioning model). Since the new model has expanded with more training data, its recognition ability is much better than that of the basic positioning model, and its feature representation is also more excellent.

[0097] S105, the updated localization model is used to group the multiple difficult case data to obtain multiple second data groups, wherein each second data group contains difficult case data corresponding to the same speaker identified by the updated localization model.

[0098] In this embodiment of the application, the audio features of the audio data and the video features of the video data in each difficult case data can be extracted using the updated localization model. The difficult case data of the same speaker can be located based on the audio features and video features. Then, the difficult case data corresponding to the same speaker can be divided into the same second data group.

[0099] S106, for each second data group, based on the audio data and video data in the second data group, determine the target difficult case data from multiple difficult case data in the second data group.

[0100] In this embodiment, for each second data group, target difficult data can be further filtered out from the second data group based on the audio and video data contained in each difficult data in the second data group, while other difficult data are considered good data, which are mistakenly identified as difficult data due to the poor recognition effect of the basic localization model. This improves the accuracy of difficult data mining.

[0101] The specific method for determining the target difficult data from multiple difficult data in the second data group based on the audio and video data in the second data group will be explained in detail through the following embodiments, and will not be elaborated here.

[0102] In another embodiment of this application, after S106, the target difficult case data can be labeled to help users understand which test data in the test dataset are the finally mined difficult case data.

[0103] Furthermore, in yet another embodiment of this application, the following steps may be included after S106:

[0104] The difficult case data in the second data group, excluding the target difficult case data, are identified as target good case data. The updated localization model is then optimized using the target good case data to obtain the target localization model.

[0105] In this embodiment, difficult example data other than the target difficult example data can be identified as target good example data, and the target good example data can be expanded into the training set of the updated localization model. The updated localization model can be retrained using the expanded training set to further optimize the updated localization model and obtain the target localization model, thereby further improving the model's recognition ability.

[0106] Furthermore, in yet another embodiment of this application, the following steps may be included after S106:

[0107] The system displays the target difficult example data through a visualization component; receives instruction information input by the user based on the target difficult example data; and annotates the noisy data when the instruction information indicates that the target difficult example data is noisy data.

[0108] In this embodiment, to further improve the accuracy of difficult case data mining, after obtaining the target difficult case data, the target difficult case data can be displayed to the user through a visualization component for further review by the user. The target difficult case data indicated by the user as noisy data is taken as the final difficult case data, thereby further improving the accuracy of difficult case data mining. The data is also labeled with the corresponding noisy data for easy viewing by the user.

[0109] In this embodiment, firstly, a basic positioning model and its corresponding test dataset are obtained. Then, multiple test data points in the test dataset are grouped using the basic positioning model to obtain multiple first data groups. Next, for each first data group, good and difficult examples are determined from the multiple test data points based on the audio data within that first data group. The basic positioning model is then optimized using the good examples to obtain an updated positioning model. Subsequently, the multiple difficult examples are grouped using the updated positioning model to obtain multiple second data groups. Finally, for each second data group, target difficult examples are determined from the multiple difficult examples within that second data group based on the audio and video data within it. This achieves intelligent mining of difficult examples, solving the current problem of the difficulty in mining difficult examples.

[0110] See Figure 2 This is a flowchart illustrating another embodiment of the difficult example mining method provided in this application. Figure 2 The process shown above Figure 1 Based on the illustrated process, this section describes how to determine the target difficult case data from multiple difficult case data in the second data group, based on the audio and video data in the second data group. For example... Figure 2 As shown, the process may include the following steps:

[0111] S201, for each difficult example data in the second data group, calculate the similarity between the audio data in the difficult example data and each other audio data to obtain at least one second similarity value; and calculate the similarity between the video data in the difficult example data and each other video data to obtain at least one third similarity value.

[0112] S202, calculate the average value corresponding to at least one of the second similarity values, and use the average value as the second similarity score corresponding to the difficult example data; and calculate the average value corresponding to at least one of the third similarity values, and use the average value as the third similarity score corresponding to the difficult example data.

[0113] S203, based on the second similarity score and the third similarity score corresponding to each difficult example data, determine the target difficult example data from the multiple difficult example data in the second data group.

[0114] The following provides a unified explanation of S201-S203:

[0115] In this embodiment, for each first data group, firstly, based on the audio features of each audio data in the data group, the second similarity score corresponding to each audio data is calculated. Specifically, for each audio data, the cosine similarity value (i.e., the second similarity value) between the audio features of the audio data and the audio features of each other audio data is determined, and all the cosine similarity values ​​corresponding to the audio data are summed and averaged to obtain its corresponding average cosine similarity xi (i = 1, 2, 3... n) (i.e., the second similarity score).

[0116] Then, based on the video features of each video data in the data group, the third similarity score corresponding to each video data is calculated. Specifically, for each video data, the cosine similarity value (i.e., the third similarity value) between the video features of the video data and the video features of each other video data is determined, and all the cosine similarity values ​​corresponding to the video data are summed and averaged to obtain its corresponding average cosine similarity yi (i = 1, 2, 3... n) (i.e., the third similarity score).

[0117] Finally, based on the second and third similarity scores corresponding to each difficult example data, the target difficult example data is determined from multiple difficult example data in the second data group.

[0118] As one possible implementation, determining the target difficult example data from multiple difficult example data in the second data group based on the second similarity score and the third similarity score corresponding to each difficult example data may include the following steps:

[0119] The second and third similarity scores corresponding to each difficult example data are input into the support vector machine, so that the support vector machine can filter out the target difficult example data from multiple difficult example data based on the second and third similarity scores corresponding to each difficult example data.

[0120] In this scheme, the second and third similarity scores corresponding to each difficult example are used as features of SVM (support vector machines). An SVM classifier is used for classification, and an optimal decision boundary is found in the two-dimensional score space, thereby further filtering out the target difficult example from the difficult example data.

[0121] As another possible implementation, determining the target difficult example data from multiple difficult example data in the second data group based on the second similarity score and the third similarity score corresponding to each difficult example data may include the following steps:

[0122] For each difficult example data, the second similarity score and the third similarity score corresponding to the difficult example data are weighted and summed to obtain the target similarity score. Difficult example data whose target similarity score is less than the second preset threshold are determined as the target difficult example data.

[0123] In this scheme, firstly, users can pre-set the first weight corresponding to the second similarity score and the second weight corresponding to the third similarity score according to their actual needs. Then, the target similarity score for each difficult example is calculated using the formula: Target Similarity Score = Second Similarity Score * First Weight + Third Similarity Score * Second Weight. Finally, difficult examples with target similarity scores less than a second preset threshold are identified as target difficult examples. This further filters target difficult examples from the difficult example data.

[0124] pass Figure 2 The process shown can further identify target difficult cases from multiple difficult cases in the second data group based on the audio and video data in the second data group, thereby further improving the accuracy of difficult case data mining.

[0125] Based on the same technical concept, this application also provides a difficult excavation device, such as... Figure 3 As shown, the device includes:

[0126] The acquisition module 301 is used to acquire a basic positioning model and a test dataset corresponding to the basic positioning model, wherein the basic positioning model is used to locate the speaker, and the test dataset contains multiple test data, each of which contains audio data and video data;

[0127] The first grouping module 302 is used to group multiple test data using the basic positioning model to obtain multiple first data groups, wherein each first data group contains test data corresponding to the same speaker identified by the basic positioning model.

[0128] The first determining module 303 is used to determine good sample data and difficult sample data from multiple test data in the first data group based on the audio data in the first data group for each first data group;

[0129] Optimization module 304 is used to optimize the basic positioning model using the good case data to obtain an updated positioning model;

[0130] The second grouping module 305 is used to group multiple difficult case data using the updated localization model to obtain multiple second data groups, wherein each second data group contains difficult case data corresponding to the same speaker identified by the updated localization model.

[0131] The second determining module 306 is used to determine target difficult case data from multiple difficult case data in the second data group based on the audio data and video data in the second data group for each second data group.

[0132] In one possible implementation, the first determining module is further configured to:

[0133] For each audio data in the first data group, the similarity between the audio data and each other audio data is calculated to obtain at least one first similarity value;

[0134] Calculate the average value corresponding to at least one of the first similarity values, and use the average value as the first similarity score corresponding to the audio data;

[0135] Audio data with a first similarity score greater than or equal to a first preset threshold are identified as good example audio, and audio data with a first similarity score less than the first preset threshold are identified as difficult example audio.

[0136] The test data corresponding to the good example audio is determined as good example data, and the test data corresponding to the difficult example audio is determined as difficult example data.

[0137] In one possible implementation, the second determining module is further configured to:

[0138] For each difficult example data in the second data group, the similarity between the audio data in the difficult example data and each other audio data is calculated to obtain at least one second similarity value; and the similarity between the video data in the difficult example data and each other video data is calculated to obtain at least one third similarity value.

[0139] Calculate the average value corresponding to at least one of the second similarity values, and use the average value as the second similarity score corresponding to the difficult example data; and calculate the average value corresponding to at least one of the third similarity values, and use the average value as the third similarity score corresponding to the difficult example data.

[0140] Based on the second and third similarity scores corresponding to each difficult example data, the target difficult example data is determined from multiple difficult example data in the second data group.

[0141] In one possible implementation, the second determining module is further configured to:

[0142] The second and third similarity scores corresponding to each difficult example data are input into the support vector machine, so that the support vector machine can filter out the target difficult example data from multiple difficult example data based on the second and third similarity scores corresponding to each difficult example data.

[0143] In one possible implementation, the second determining module is further configured to:

[0144] For each difficult example data, the second similarity score and the third similarity score corresponding to the difficult example data are weighted and summed to obtain the target similarity score;

[0145] Difficult example data whose corresponding target similarity score is less than the second preset threshold is identified as the target difficult example data.

[0146] In one possible implementation, the optimization module is further configured to:

[0147] The difficult case data in the second data group, excluding the target difficult case data, are identified as the target good case data;

[0148] The updated localization model is optimized using the target good case data to obtain the target localization model.

[0149] In one possible implementation, the device further includes a labeling module for:

[0150] The target difficult example data is displayed through visualization components;

[0151] Receive instruction information input by the user based on the target difficulty example data;

[0152] If the indication information indicates that the target difficult example data is noisy data, the noisy data is labeled.

[0153] In this embodiment, firstly, a basic positioning model and its corresponding test dataset are obtained. Then, multiple test data points in the test dataset are grouped using the basic positioning model to obtain multiple first data groups. Next, for each first data group, good and difficult examples are determined from the multiple test data points based on the audio data within that first data group. The basic positioning model is then optimized using the good examples to obtain an updated positioning model. Subsequently, the multiple difficult examples are grouped using the updated positioning model to obtain multiple second data groups. Finally, for each second data group, target difficult examples are determined from the multiple difficult examples within that second data group based on the audio and video data within it. This achieves intelligent mining of difficult examples, solving the current problem of the difficulty in mining difficult examples.

[0154] Based on the same technical concept, embodiments of this application also provide an electronic device, such as... Figure 4 As shown, it includes a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0155] Memory 113 is used to store computer programs;

[0156] When processor 111 executes a program stored in memory 113, it performs the following steps:

[0157] Obtain a basic localization model and a test dataset corresponding to the basic localization model, wherein the basic localization model is used to locate the speaker, and the test dataset contains multiple test data, each of which contains audio data and video data;

[0158] The basic localization model is used to group multiple test data to obtain multiple first data groups, wherein each first data group contains test data corresponding to the same speaker identified by the basic localization model;

[0159] For each first data group, good example data and difficult example data are determined from multiple test data in the first data group based on the audio data in the first data group;

[0160] The base positioning model is optimized using the good case data to obtain an updated positioning model;

[0161] The updated localization model is used to group multiple difficult case data to obtain multiple second data groups, wherein each second data group contains difficult case data corresponding to the same speaker identified by the updated localization model;

[0162] For each second data group, based on the audio and video data in the second data group, target difficult data is determined from multiple difficult data in the second data group.

[0163] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0164] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0165] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0166] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0167] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described hard case mining methods.

[0168] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the hard case mining methods described in the above embodiments.

[0169] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0170] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0171] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0172] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A difficult case mining method characterized by comprising: The method includes: Obtain a basic localization model and a test dataset corresponding to the basic localization model, wherein the basic localization model is used to locate the speaker, and the test dataset contains multiple test data, each of which contains audio data and video data; The basic localization model is used to group multiple test data to obtain multiple first data groups, wherein each first data group contains test data corresponding to the same speaker identified by the basic localization model; For each first data group, good example data and difficult example data are determined from multiple test data in the first data group based on the audio data in the first data group; The base positioning model is optimized using the good case data to obtain an updated positioning model; The updated localization model is used to group multiple difficult case data to obtain multiple second data groups, wherein each second data group contains difficult case data corresponding to the same speaker identified by the updated localization model; For each second data group, based on the audio and video data in the second data group, target difficult data is determined from multiple difficult data in the second data group.

2. The method according to claim 1, characterized in that, The step of determining good and difficult test data from multiple test data in the first data group based on audio data in the first data group includes: For each audio data in the first data group, the similarity between the audio data and each other audio data is calculated to obtain at least one first similarity value; Calculate the average value corresponding to at least one of the first similarity values, and use the average value as the first similarity score corresponding to the audio data; Audio data with a first similarity score greater than or equal to a first preset threshold are identified as good example audio, and audio data with a first similarity score less than the first preset threshold are identified as difficult example audio. The test data corresponding to the good example audio is determined as good example data, and the test data corresponding to the difficult example audio is determined as difficult example data.

3. The method of claim 1, wherein, The step of determining the target difficult case data from multiple difficult case data in the second data group based on the audio and video data in the second data group includes: For each difficult example data in the second data group, the similarity between the audio data in the difficult example data and each other audio data is calculated to obtain at least one second similarity value; and the similarity between the video data in the difficult example data and each other video data is calculated to obtain at least one third similarity value. Calculate the average value corresponding to at least one of the second similarity values, and use the average value as the second similarity score corresponding to the difficult example data; and calculate the average value corresponding to at least one of the third similarity values, and use the average value as the third similarity score corresponding to the difficult example data. Based on the second and third similarity scores corresponding to each difficult example data, the target difficult example data is determined from multiple difficult example data in the second data group.

4. The method of claim 3, wherein, The step of determining the target difficult example data from multiple difficult example data in the second data group based on the second similarity score and the third similarity score corresponding to each difficult example data includes: The second and third similarity scores corresponding to each difficult example data are input into the support vector machine, so that the support vector machine can filter out the target difficult example data from multiple difficult example data based on the second and third similarity scores corresponding to each difficult example data.

5. The method of claim 3, wherein, The step of determining the target difficult example data from multiple difficult example data in the second data group based on the second similarity score and the third similarity score corresponding to each difficult example data includes: For each difficult example data, the second similarity score and the third similarity score corresponding to the difficult example data are weighted and summed to obtain the target similarity score; Difficult example data whose corresponding target similarity score is less than the second preset threshold is identified as the target difficult example data.

6. The method of claim 1, wherein, After determining the target difficult case data from multiple difficult case data in the second data group based on the audio and video data in the second data group, the method further includes: The difficult case data in the second data group, excluding the target difficult case data, are identified as the target good case data; The updated localization model is optimized using the target good case data to obtain the target localization model.

7. The method of claim 1, wherein, After determining the target difficult case data from multiple difficult case data in the second data group based on the audio and video data in the second data group, the method further includes: The target difficult example data is displayed through visualization components; Receive instruction information input by the user based on the target difficulty example data; If the indication information indicates that the target difficult example data is noisy data, the noisy data is labeled.

8. A difficult case mining apparatus characterized by comprising: The device includes: The acquisition module is used to acquire a basic positioning model and a test dataset corresponding to the basic positioning model. The basic positioning model is used to locate the speaker, and the test dataset contains multiple test data, each of which contains audio data and video data. The first grouping module is used to group multiple test data using the basic positioning model to obtain multiple first data groups, wherein each first data group contains test data corresponding to the same speaker identified by the basic positioning model; The first determining module is used to determine good sample data and difficult sample data from multiple test data in the first data group based on the audio data in the first data group for each first data group; The optimization module is used to optimize the basic positioning model using the good case data to obtain an updated positioning model; The second grouping module is used to group multiple difficult case data using the updated localization model to obtain multiple second data groups, wherein each second data group contains difficult case data corresponding to the same speaker identified by the updated localization model; The second determining module is used to determine target difficult data from multiple difficult data in the second data group based on the audio data and video data in the second data group for each second data group.

9. An electronic device, comprising: It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the hard case mining method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the difficult case mining method according to any one of claims 1-7.

Images