Training data set construction method and apparatus, electronic device, and computer readable medium

Abstract

The present disclosure provides a training data set construction method and device, electronic equipment and computer readable medium, relating to the technical field of data processing. The method comprises: obtaining a basic data set, constructing a first data set according to the semantic identification of the training data, constructing a second / third data set according to the semantic similarity between training data with the same / different semantic identification, and constructing a target data set. The embodiment of the present disclosure reorganizes the data in the basic data set according to the similarity between the data and the data semantic identification according to the preset rules, respectively forms the first, second and third data sets, and combines the second and / or third data set with the first data set to construct the target data set. The data complexity in the target data set is high, the data set acquisition cost is low, and the training model can be trained efficiently and accurately by using the data set.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, and in particular, the present disclosure relates to a training data set construction method and device, electronic equipment and computer readable medium. BACKGROUND

[0002] With the rapid development of computer technology, algorithm models play an important role in many places, especially in the field of intelligent identification, algorithm models are often used as conventional and basic tools.

[0003] In the existing model construction scheme, a data set is generally used to train the model to improve the quality of the model, and the quality of the data set will directly affect the quality of the final model, so the selection of the data set is very important. In the field of text analysis, high-quality data sets are generally difficult to obtain and have high costs, while low-quality data sets have low costs but result in low robustness of the trained model, leading to insufficient model quality.

[0004] Therefore, in the prior art, the selection of the data set cannot obtain high-quality data sets at low cost, resulting in low quality of the trained model. SUMMARY

[0005] The purpose of the present disclosure is to at least solve one of the above technical defects, in particular, the selection of the data set in the prior art cannot obtain high-quality data sets at low cost, resulting in low quality of the trained model.

[0006] In a first aspect, a training data set construction method is provided, the method comprising:

[0007] obtaining a basic data set, the basic data set comprising a plurality of training data and a semantic identifier corresponding to each training data;

[0008] constructing a first data set according to the semantic identifier of the training data, the first data set comprising a first training data pair with a first label and a second training data pair with a second label, wherein the first label represents that the semantic identifiers of the two training data contained in the first training data pair are the same, and the second label represents that the semantic identifiers of the two training data contained in the second training data pair are different;

[0009] Based on the semantic identifiers and text similarity between training data in each of the aforementioned basic datasets, a second dataset and / or a third dataset are constructed; wherein, the second dataset includes a third training data pair with a first label and a semantic similarity lower than a first preset threshold; the third dataset includes a fourth training data pair with a second label and a semantic similarity higher than a second preset threshold; wherein, the second preset threshold is greater than the first preset threshold;

[0010] A target dataset is constructed based on the second dataset and / or the third dataset, as well as the first dataset, and the target dataset is used for model training.

[0011] As one possible implementation of this disclosure, the step of constructing the first dataset based on the semantic identifiers of the training data includes:

[0012] For any of the training data, a third preset number of training data in the basic dataset that have the same semantic identifier as the training data are respectively combined with the training data to form the first training data pair, and the first label is marked.

[0013] For any of the training data, a fourth preset number of training data in the base dataset that have different semantic identifiers from the training data are respectively combined with the training data to form the second data pair, and labeled with the second label.

[0014] As one possible implementation of this disclosure, the step of constructing a second dataset based on the semantic identifiers of each training data item in each of the aforementioned basic datasets and the textual similarity between the data items includes:

[0015] Calculate the first semantic similarity between training data with the same semantic identifier in the basic dataset;

[0016] Select training data pairs whose semantic similarity is less than the first set value as the third data pair, and label them with the first label; or,

[0017] Based on the ascending order of each of the first semantic similarities, select the first preset number of semantic similarity training data pairs that rank highest as the third data pair and label them with the first tag.

[0018] As one possible implementation of this disclosure, the step of constructing a third dataset based on the semantic identifiers of each training data item in each of the aforementioned basic datasets and the textual similarity between the data items includes:

[0019] Calculate the second semantic similarity between training data with different semantic identifiers in the basic dataset;

[0020] Select training data pairs whose second semantic similarity is greater than the second set value as the fourth data pair, and label them with the second label; or,

[0021] Based on the descending order of the second semantic similarity scores, a second preset number of semantic similarity training data pairs that rank highest are selected as the fourth data pair and labeled with the second tag.

[0022] As one possible implementation of this disclosure, the semantic similarity between the training data can be calculated using an optimal matching similarity algorithm.

[0023] Secondly, a training dataset construction apparatus is provided, the apparatus comprising:

[0024] The basic dataset acquisition module is used to acquire the basic dataset, which includes multiple training data and semantic identifiers corresponding to each training data.

[0025] The first dataset construction module is used to construct a first dataset based on the semantic identifier of the training data. The first dataset includes a first training data pair with a first label and a second training data pair with a second label. The first label indicates that the two training data pairs contained in the first training data pair have the same semantic identifier, and the second label indicates that the two training data pairs contained in the second training data pair have different semantic identifiers.

[0026] The second and third dataset construction modules are used to construct a second dataset and / or a third dataset based on the semantic identifiers of each training data in each of the basic datasets and the text similarity between the data; wherein, the second dataset includes a third training data pair with a first label and a semantic similarity lower than a first preset threshold; the third dataset includes a fourth training data pair with a second label and a semantic similarity higher than a second preset threshold; wherein, the second preset threshold is greater than the first preset threshold;

[0027] A target dataset construction module is used to construct a target dataset based on the second dataset and / or the third dataset, as well as the first dataset, the target dataset being invoked for model training.

[0028] As one possible implementation of this disclosure, the first dataset construction module includes:

[0029] The first labeling unit is used to, for any training data, form the first training data pair with the training data by combining a third preset number of training data in the basic dataset that have the same semantic identifier as the training data, and label the training data with the first label.

[0030] The second labeling unit is used to, for any training data, form the second data pair with a fourth preset number of training data in the base dataset that have different semantic identifiers from the training data, and label them with the second label.

[0031] Thirdly, an electronic device is provided, the electronic device comprising:

[0032] One or more processors;

[0033] Memory;

[0034] One or more applications, wherein the applications are stored in memory and configured to be executed by one or more processors, the applications being configured to: execute the training dataset construction method described above.

[0035] Fourthly, a computer-readable medium is provided, which stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the above-described training dataset construction method.

[0036] This embodiment of the disclosure reorganizes the data in the basic dataset according to the text similarity between the data and whether the semantic identifiers of the data are the same, according to preset rules, to form a first dataset, a second dataset, and a third dataset. Then, at least one of the second and third datasets is merged with the first dataset to construct a target dataset. The target dataset has high data complexity and low acquisition cost. Using this dataset, an efficient and accurate training model can be trained. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments of this disclosure will be briefly introduced below.

[0038] Figure 1 A flowchart illustrating a method for constructing a training dataset according to an embodiment of this disclosure;

[0039] Figure 2 A flowchart illustrating a method for constructing a second dataset provided in an embodiment of this disclosure;

[0040] Figure 3 A flowchart illustrating a method for constructing a third dataset provided in this embodiment of the disclosure;

[0041] Figure 4 This is a schematic diagram of the structure of a training dataset construction apparatus provided in an embodiment of the present disclosure;

[0042] Figure 5 This is a schematic diagram of the structure of a first dataset construction module provided in an embodiment of the present disclosure;

[0043] Figure 6 This is a schematic diagram of the structure of a second dataset and a third dataset construction module provided in an embodiment of this disclosure;

[0044] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.

[0045] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. Detailed Implementation

[0046] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0047] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0048] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0049] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish between devices, modules or units, and are not intended to limit these devices, modules or units to necessarily be different devices, modules or units, nor are they intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0050] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0051] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0052] The technical solutions of this disclosure and how they solve the aforementioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this disclosure will now be described with reference to the accompanying drawings.

[0053] This disclosure provides a method for constructing a training dataset, used to construct a dataset for model training, such as... Figure 1 As shown, the method includes:

[0054] Step S101: Obtain the basic dataset, which includes multiple training data and semantic identifiers corresponding to each training data.

[0055] Step S102: Construct a first dataset based on the semantic identifiers of the training data. The first dataset includes a first training data pair with a first label and a second training data pair with a second label. The first label indicates that the two training data pairs contained in the first training data pair have the same semantic identifier, and the second label indicates that the two training data pairs contained in the second training data pair have different semantic identifiers.

[0056] Step S103: Based on the semantic identifiers of each training data in each of the basic datasets and the textual similarity between the data, construct a second dataset and / or a third dataset; wherein, the second dataset includes a third training data pair with a first label and a semantic similarity lower than a first preset threshold; the third dataset includes a fourth training data pair with a second label and a semantic similarity higher than a second preset threshold; wherein, the second preset threshold is greater than the first preset threshold.

[0057] Step S104: Construct a target dataset based on the second dataset and / or the third dataset, and the first dataset, the target dataset being used for model training.

[0058] The training dataset construction method provided in this disclosure is used to construct a training dataset, namely the aforementioned target dataset. As described, the target dataset includes both data pairs with the same semantic identifier and data pairs with different semantic identifiers. Therefore, the target dataset obtained based on the training dataset construction method provided in this disclosure can be used for training a data classification model or a semantic similarity discrimination model. For example, if the training dataset is a text dataset, the constructed target dataset can be used for training a text similarity discrimination model.

[0059] The optional embodiments of each of the above steps are described below.

[0060] In step S101, a basic dataset is obtained, which includes multiple training data and semantic identifiers corresponding to each training data.

[0061] In this embodiment of the disclosure, for ease of explanation, the construction of a text training dataset is taken as an example. The basic dataset is the basic classification dataset in this embodiment of the disclosure, that is, a dataset with semantic labeling. The method of obtaining the basic dataset is not limited in this embodiment of the disclosure. For example, the basic dataset can be an existing commonly used similarity dataset or classification dataset, which contains a lot of data with semantic labeling.

[0062] Optionally, the semantic identifier of the training data can refer to the semantic identifier of the training data. For example, the text data "I can't upload the video I made" and the text data "The video won't let me upload" are two pieces of data with the same semantic identifier.

[0063] Of course, in practical applications, the specific classification criteria for data semantic identification can be configured according to actual application needs, and this embodiment does not impose any limitations. In this embodiment, the labels of each training data in the basic dataset can be manually labeled. If the basic dataset is an existing classification dataset, the semantic identification of the training data can be the semantic identification label inherent to each data in that classification dataset.

[0064] In this embodiment of the disclosure, the basic dataset is text data that is relatively easy to obtain and has low data complexity. Each piece of text data includes text information and a text semantic identifier used to represent the semantics of the text information.

[0065] In step S102, a first dataset is constructed based on the semantic identifiers of the training data. The first dataset includes a first training data pair with a first label and a second training data pair with a second label. The first label indicates that the two training data pairs contained in the first training data pair have the same semantic identifier, and the second label indicates that the two training data pairs contained in the second training data pair have different semantic identifiers.

[0066] In this embodiment of the disclosure, the first dataset refers to the first dataset after the basic dataset is reorganized according to preset rules. This dataset contains two types of data pairs: first training data pairs with the same semantic label and second training data pairs with different semantic labels.

[0067] For the purposes of this embodiment, n pairs of similar data are randomly selected from the basic dataset based on the semantic identifiers of the data. (It can be understood that similar data pairs refer to data pairs with a similarity greater than a threshold, i.e., data pairs with the same semantic identifier.) Taking one data pair A1 as an example, data pair A1 is (text a1, text a2, similar), where text a1 and text a2 are the text information of the data, and similarity is the first label of text a1 and text a2. Specifically, data pair A1 is (I watch the live stream and it's choppy, my live stream video is choppy, similar). M pairs of different data are randomly selected from the basic dataset based on the semantic identifiers of the data. Taking one data pair A2 as an example, data pair A2 is (text a3, text a4, different), where text a3 and text a4 are the text information of the data, and differentness is the second label of text a3 and text a4. Specifically, data pair A2 is (I watch the live stream and it's choppy, I start the live stream and it's choppy, different). The above n similar data pairs and m different data pairs are used together to construct the first dataset.

[0068] In step S103, a second dataset and / or a third dataset are constructed based on the semantic identifiers of each training data in each of the basic datasets and the textual similarity between the data; wherein, the second dataset includes a third training data pair with a first label and a semantic similarity lower than a first preset threshold; the third dataset includes a fourth training data pair with a second label and a semantic similarity higher than a second preset threshold; wherein, the second preset threshold is greater than the first preset threshold.

[0069] In this embodiment of the disclosure, both the second dataset and the third dataset are constructed based on the similarity between training data. Specifically, the second dataset is constructed based on the semantic similarity between training data with the same semantic identifier, and the third dataset is constructed based on the similarity between training data with different semantic identifiers.

[0070] In the embodiments of this disclosure, when constructing the second dataset, the similarity between two text data in training data pairs with the same semantic identifier can be calculated. Then, the second dataset is constructed according to the ranking of similarity or by taking a preset number of data pairs according to a threshold. Specifically, if there are x pairs of data pairs with similar semantic identifiers, the similarity between the two training data in each of the x pairs of data pairs is calculated. The x pairs of data pairs are then sorted according to the similarity, and a preset number of data pairs with lower similarity are constructed as the second dataset. Similarly, when constructing the third dataset, the similarity between two text data in training data pairs with different semantic identifiers can be calculated. Then, the third dataset is constructed according to the ranking of similarity or by taking a preset number of data pairs according to a threshold. Specifically, if there are y pairs of data pairs with different semantic identifiers, the similarity between the two training data in each of the y pairs of data pairs is calculated. The y pairs of data pairs are then sorted according to the similarity, and a preset number of data pairs with higher similarity are constructed as the third dataset.

[0071] In step S104, a target dataset is constructed based on the second dataset and / or the third dataset, as well as the first dataset, and the target dataset is used for model training.

[0072] In this embodiment of the disclosure, after the first, second, and third datasets are constructed, a target dataset is constructed based on the first, second, and third datasets.

[0073] In the embodiments of this disclosure, when constructing the target dataset, data pairs from the first dataset, the second dataset, and the third dataset are merged into one dataset to construct the third dataset.

[0074] This embodiment of the disclosure reorganizes the data in the basic dataset according to the similarity between the data and whether the semantic identifiers of the data are the same, according to preset rules, to form a first dataset, a second dataset, and a third dataset. Then, the second and / or third datasets are merged with the first dataset to construct a target dataset. The basic dataset is easy to obtain and the dataset acquisition cost is low. The target dataset contains data pairs with high similarity but not the same type of data and data pairs with low similarity but the same type of data. Using this dataset, an efficient and accurate training model can be trained.

[0075] This disclosure provides a possible implementation method in which the construction of the first dataset based on the semantic identifiers of the training data includes:

[0076] For any of the training data, a third preset number of training data in the basic dataset that have the same semantic identifier as the training data are respectively combined with the training data to form the first training data pair, and the first label is marked.

[0077] For any of the training data, a fourth preset number of training data in the base dataset that have different semantic identifiers from the training data are respectively combined with the training data to form the second data pair, and labeled with the second label.

[0078] In this embodiment of the disclosure, when constructing the first dataset, it is necessary to construct two data pairs based on the semantic identifiers of the training data in the basic dataset. Each data pair contains two text data and the labels of the two text data. If the two text data in the first data pair have similar semantic identifiers, then the first label is "similar". If the two text data in the second data pair have different semantic identifiers, then the second label is "different".

[0079] In this embodiment of the disclosure, n pairs of similar data are randomly selected from the basic dataset based on the semantic identifiers of the data (it can be understood that similar data pairs refer to data pairs with a similarity greater than a threshold, and similar data pairs are data pairs with the same semantic identifier). Taking one data pair A1 as an example, data pair A1 is (text a1, text a2, similar), where text a1 and text a2 are the text information of the data, and similarity is the first label of text a1 and text a2. Specifically, for example, data pair A1 is (I watch the live stream is choppy, my live stream video is choppy, similar). Based on the semantic identifiers of the data in the basic dataset, m pairs of different data are randomly selected from the basic dataset. Taking one data pair A2 as an example, data pair A2 is (text a3, text a4, different), where text a3 and text a4 are the text information of the data, and differentness is the second label of text a3 and text a4. Specifically, for example, data pair A2 is (I watch the live stream is choppy, I start the live stream is choppy, different). The above n similar data pairs and m different data pairs are used together to construct the first dataset.

[0080] This embodiment of the disclosure groups training data with the same semantic identifier and different semantic identifiers into pairs and labels the data pairs to construct a first dataset. The construction of the first dataset can increase the complexity of the target dataset.

[0081] This disclosure provides a possible implementation, in which, as shown in the embodiments, Figure 2 As shown, based on the semantic identifiers of each training data point in each base dataset and the textual similarity between data points, the semantic similarity of the second dataset is constructed, including:

[0082] Step S210: Calculate the first semantic similarity between training data with the same semantic identifier in the basic dataset;

[0083] Step S211: Select the training data pair with the first semantic similarity less than the first set value as the third data pair, and label it with the first tag; or

[0084] Step S212: Based on the ascending order of each of the first semantic similarities, select the first preset number of semantic similarity training data pairs that are ranked first, and use them as the third data pairs and label them with the first tag.

[0085] In this embodiment of the disclosure, when constructing the second dataset, the similarity between each training data with the same semantic identifier in the basic dataset is calculated separately, and the BM (best match) similarity algorithm can be used for the calculation.

[0086] For the purposes of this disclosure, for ease of explanation, the BM similarity algorithm is used to calculate the similarity between pairs of training data for training data with the same semantic identifier. For example, if there are training data b1, b2, b3, b4, b5, and b6 with the first semantic identifier in the basic dataset, the similarity between each pair of training data texts can be calculated by the algorithm, resulting in 15 data pairs and the similarity of the text data in the 15 data pairs.

[0087] In this embodiment of the disclosure, after calculating the similarity of training data with similar semantic labels in the basic dataset, a preset number of data pairs are constructed into a third data pair, and a first label is assigned to them as similar. Specifically, as in the above embodiment, the 15 data pairs can be sorted according to their similarity, and a preset number of data pairs with the highest similarity are selected as the third data pair. Alternatively, data pairs with similarity exceeding a preset threshold can be selected as the third data pair and assigned a first label to them as similar. The set of third data pairs formed above constitutes a portion of the second dataset.

[0088] This disclosure embodiment calculates the similarity between training data with the same semantic identifier in the basic dataset and constructs corresponding data pairs. Based on the similarity, data pairs that meet the conditions are selected to construct a second dataset, thereby increasing the complexity of the target dataset.

[0089] This disclosure provides a possible implementation, in which, as shown in the embodiments, Figure 3 As shown, a third dataset is constructed based on the semantic identifiers of each data point in each base dataset and the textual similarity between data points. Semantic similarity includes:

[0090] Step S310: Calculate the second semantic similarity between training data with different semantic identifiers in the basic dataset;

[0091] Step S311: Select training data pairs with a second semantic similarity greater than the second set value as the fourth data pair, and label them with the second tag; or,

[0092] Step S312: Based on the descending order of the second semantic similarity scores, select the top-ranked second preset number of semantic similarity training data pairs as the fourth data pair and label them with the second tag.

[0093] In this embodiment of the disclosure, when constructing the second dataset, the similarity between each training data with different semantic identifiers in the basic dataset is calculated separately, and the BM similarity algorithm can be used for the calculation.

[0094] For the purposes of this disclosure, the BM similarity algorithm is used to calculate the similarity between pairs of training data for training data with different semantic identifiers. For example, if there are training data c1, c2, c3, c4, c5, and c6 with different semantic identifiers in the basic dataset, the similarity between each pair of training data texts can be calculated by the algorithm, resulting in 15 data pairs and the similarity between the text data in the 15 data pairs.

[0095] In this embodiment of the disclosure, after calculating the similarity of training data with different semantic identifiers in the basic dataset, a preset number of data pairs are constructed into a fourth data pair according to a preset second condition, and the second label is marked as different. Specifically, as in the 15 data pairs in the above embodiment, a preset number of data pairs can be selected as the fourth data pair based on their similarity, or data pairs with similarity exceeding a preset threshold can be selected as the fourth data pair and marked as different. The set of fourth data pairs formed above constitutes a portion of the data in the third dataset.

[0096] This disclosure embodiment calculates the similarity between training data with different semantic identifiers in the basic dataset and constructs corresponding data pairs. Based on the similarity, data pairs that meet the conditions are selected to construct a second dataset, thereby increasing the complexity of the target dataset.

[0097] In this embodiment of the disclosure, taking the above embodiment as an example, when constructing the second dataset using the third data pair, the third data pair used to construct the second dataset needs to meet a first condition. The first condition may be that the semantic similarity between two training data is less than a first preset value and / or the semantic similarity between two data is the last n in the semantic similarity ranking of all data pairs with the same semantic identifier.

[0098] In this embodiment of the disclosure, the first condition is that the semantic similarity between two training data sets is less than a first preset value. When selecting a third data pair to construct the second dataset, the data pairs with similarity less than the first preset value are determined as the third data pairs based on the similarity of training data pairs with the same semantic identifier in the basic dataset. Specifically, the first preset value can be 0.5, meaning that training data pairs with similar semantic identifiers and similarity less than 0.5 are the third data pairs. In another embodiment provided by this disclosure, the first condition is that the semantic similarity between two data sets is among the last n in the semantic similarity ranking of all data pairs with the same semantic identifier. When selecting a third data pair to construct the second dataset, the similarity of training data pairs with similar semantic identifiers in the basic dataset is sorted, and n data pairs with lower similarity are selected in order to be determined as the third data pairs for constructing the second dataset.

[0099] In this embodiment of the disclosure, a third data pair is determined by selecting data pairs with lower similarity from data pairs with similar semantic identifiers to construct a second dataset. When the model is trained using the dataset formed by data with lower similarity but similar semantic identifiers, the model can be enhanced to recognize identical text pairs with lower similarity, thereby improving the accuracy of model recognition.

[0100] In this embodiment of the disclosure, taking the above embodiment as an example, when constructing the third dataset using the fourth data pair, the fourth data pair used to construct the third dataset needs to meet a second condition. The second condition may be that the semantic similarity between two training data is greater than a second preset value and / or the semantic similarity between two data is among the top m in the semantic similarity ranking of all data pairs with different semantic identifiers.

[0101] In this embodiment of the disclosure, the second condition is that the semantic similarity between two training data pairs is greater than a second preset value. When selecting a fourth data pair to construct the third dataset, the data pairs with a similarity greater than the second preset value are determined as the fourth data pairs based on the similarity of training data pairs with different semantic identifiers in the basic dataset. Specifically, the first preset value can be 0.6, meaning that training data pairs with different semantic identifiers and a similarity greater than 0.6 are the fourth data pairs. In another embodiment provided by this disclosure, the second condition is that the semantic similarity between two data pairs is among the top m in the semantic similarity ranking of all data pairs with different semantic identifiers. When selecting a fourth data pair to construct the third dataset, the similarity of training data pairs with different semantic identifiers in the basic dataset is sorted, and m data pairs with higher similarity are selected in order to be determined as the fourth data pairs for constructing the third dataset.

[0102] In this embodiment of the disclosure, a fourth data pair is determined by selecting a data pair with high similarity from data pairs with similar semantic identifiers, which is used to construct a third dataset. When the model is trained using a dataset formed by data with high similarity but different semantic identifiers, the model can be enhanced to recognize different text pairs with high similarity, thereby improving the accuracy of model recognition.

[0103] This embodiment of the disclosure reorganizes the data in the basic dataset according to the similarity between the data and whether the semantic identifiers of the data are the same, according to preset rules, to form a first dataset, a second dataset, and a third dataset. Then, the second and / or third datasets are merged with the first dataset to construct a target dataset. The basic dataset is easy to obtain and the dataset acquisition cost is low. The target dataset contains data pairs with high similarity but not the same type of data and data pairs with low similarity but the same type of data. Using this dataset, an efficient and accurate training model can be trained.

[0104] This disclosure provides a possible implementation method in which the semantic similarity between the training data can be calculated using an optimal matching similarity algorithm.

[0105] In this embodiment of the disclosure, the best matching similarity algorithm can be used when calculating the semantic similarity between training data.

[0106] This embodiment of the disclosure reorganizes the data in the basic dataset according to the similarity between the data and whether the semantic identifiers of the data are the same, according to preset rules, to form a first dataset, a second dataset, and a third dataset respectively. Then, at least one of the second dataset and the third dataset is merged with the first dataset to construct a target dataset. The target dataset has high data complexity and low acquisition cost. Using this dataset, an efficient and accurate training model can be trained.

[0107] This disclosure provides a training dataset construction apparatus, such as... Figure 4 As shown, the training dataset construction 40 may include: a basic dataset acquisition module 401, a first dataset construction module 402, a second and third dataset construction module 403, and a target dataset construction module 404, wherein,

[0108] The basic dataset acquisition module 401 is used to acquire a basic dataset, which includes multiple training data and semantic identifiers corresponding to each training data.

[0109] The first dataset construction module 402 is used to construct a first dataset based on the semantic identifier of the training data. The first dataset includes a first training data pair with a first label and a second training data pair with a second label. The first label indicates that the two training data contained in the first training data pair have the same semantic identifier, and the second label indicates that the two training data contained in the second training data pair have different semantic identifiers.

[0110] The second and third dataset construction module 403 is used to construct a second dataset and / or a third dataset based on the semantic identifiers of each training data in each of the basic datasets and the text similarity between the data; wherein, the second dataset includes a third training data pair with a first label and a semantic similarity lower than a first preset threshold; the third dataset includes a fourth training data pair with a second label and a semantic similarity higher than a second preset threshold; wherein, the second preset threshold is greater than the first preset threshold;

[0111] The target dataset construction module 404 is used to construct a target dataset based on the second dataset and / or the third dataset, as well as the first dataset, the target dataset being invoked for model training.

[0112] Optional, such as Figure 5 As shown, the first dataset construction module 402 provided in this embodiment includes:

[0113] The first labeling unit 421 is used to, for any training data, form the first training data pair with the training data by combining a third preset number of training data in the basic dataset that have the same semantic identifier as the training data, and label the training data with the first label.

[0114] The second labeling unit 422 is used to, for any training data, form a second data pair with the training data by combining a fourth preset number of training data in the basic dataset that have different semantic identifiers from the training data, and label them with the second label.

[0115] Optional, such as Figure 6 As shown, the second and third dataset construction module 403 provided in this embodiment includes:

[0116] The first semantic similarity calculation unit 431 is used to calculate the first semantic similarity between training data with the same semantic identifier in the basic dataset;

[0117] The third data pair determination unit 432 is used to select training data pairs whose first semantic similarity is less than the first set value as the third data pair and label them with the first label; or, according to the ascending order of each of the first semantic similarities, select the first preset number of training data pairs that are ranked first as the third data pair and label them with the first label.

[0118] Optionally, the second and third dataset construction module 403 provided in this embodiment of the disclosure, when constructing the third dataset based on the semantic similarity between training data with different semantic identifiers, can be used for:

[0119] Calculate the second semantic similarity between training data with different semantic labels in the base dataset;

[0120] Select training data pairs with a second semantic similarity greater than a second set value as the fourth data pair, and label them with the second label; or,

[0121] Based on the descending order of the second semantic similarity scores, a second preset number of training data pairs that rank highly are selected as the fourth data pair and labeled with the second tag.

[0122] Optionally, when constructing the second dataset based on the semantic similarity between training data with the same semantic identifier, the semantic similarity between the training data provided in this embodiment of the disclosure can be calculated by the BM similarity algorithm.

[0123] The training dataset construction apparatus of this disclosure can execute the training dataset construction method shown in the above embodiments of this disclosure, and its implementation principle is similar, so it will not be described again here.

[0124] The following is for reference. Figure 7 The diagram illustrates a structural schematic of an electronic device 700 suitable for implementing embodiments of the present disclosure. Terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0125] The electronic device includes a memory and a processor, wherein the processor may be referred to as processing device 701 below, and the memory may include at least one of read-only memory (ROM) 702, random access memory (RAM) 703 and storage device 708 below, as detailed below:

[0126] like Figure 7 As shown, the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0127] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0128] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the computer program is executed by processing device 701, it performs the functions defined in the methods of embodiments of this disclosure.

[0129] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium, a computer-readable medium, or any combination thereof. A computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0130] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0131] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0132] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: acquire a basic dataset, the basic dataset including multiple training data and semantic identifiers corresponding to each training data;

[0133] A first dataset is constructed based on the semantic identifiers of the training data. The first dataset includes a first training data pair with a first label and a second training data pair with a second label. The first label indicates that the two training data pairs in the first training data pair have the same semantic identifier, and the second label indicates that the two training data pairs in the second training data pair have different semantic identifiers. A second dataset and / or a third dataset are constructed based on the semantic identifiers of each data pair in each of the basic datasets and the text similarity between the data pairs. The second dataset includes a third training data pair with a first label and a semantic similarity lower than a first preset threshold. The third dataset includes a fourth training data pair with a second label and a semantic similarity higher than a second preset threshold. The second preset threshold is greater than the first preset threshold. A target dataset is constructed based on the second dataset and / or the third dataset, and the first dataset. The target dataset is used for model training.

[0134] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0135] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0136] The modules or units described in the embodiments of this disclosure can be implemented in software or hardware. The names of modules or units do not necessarily limit the unit itself; for example, the first dataset construction module can also be described as "a module for constructing a dataset".

[0137] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0138] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0139] Based on one or more examples provided in this disclosure, a method for constructing a training dataset is provided, including:

[0140] Obtain the basic dataset, which includes multiple training data sets and semantic identifiers corresponding to each training data set;

[0141] A first dataset is constructed based on the semantic identifiers of the training data. The first dataset includes a first training data pair with a first label and a second training data pair with a second label. The first label indicates that the two training data pairs contained in the first training data pair have the same semantic identifier, and the second label indicates that the two training data pairs contained in the second training data pair have different semantic identifiers.

[0142] Based on the semantic identifiers and text similarity between training data in each of the aforementioned basic datasets, a second dataset and / or a third dataset are constructed; wherein, the second dataset includes a third training data pair with a first label and a semantic similarity lower than a first preset threshold; the third dataset includes a fourth training data pair with a second label and a semantic similarity higher than a second preset threshold; wherein, the second preset threshold is greater than the first preset threshold;

[0143] A target dataset is constructed based on the second dataset and / or the third dataset, as well as the first dataset, and the target dataset is used for model training.

[0144] Furthermore, a first dataset is constructed based on the semantic identifiers of the training data, including:

[0145] For any of the training data, a third preset number of training data in the basic dataset that have the same semantic identifier as the training data are respectively combined with the training data to form the first training data pair, and the first label is marked.

[0146] For any of the training data, a fourth preset number of training data in the base dataset that have different semantic identifiers from the training data are respectively combined with the training data to form the second data pair, and labeled with the second label.

[0147] Furthermore, based on the semantic identifiers of each data point in the aforementioned basic datasets and the textual similarity between data points, a second dataset semantic similarity is constructed, including:

[0148] Calculate the first semantic similarity between training data with the same semantic identifier in the basic dataset;

[0149] Select training data pairs whose semantic similarity is less than the first set value as the third data pair, and label them with the first label; or,

[0150] Based on the ascending order of each of the first semantic similarities, select the first preset number of semantic similarity training data pairs that rank highest as the third data pair and label them with the first tag.

[0151] Furthermore, based on the semantic identifiers of each data point in the aforementioned basic datasets and the textual similarity between data points, a third dataset is constructed, including:

[0152] Calculate the second semantic similarity between training data with different semantic identifiers in the basic dataset;

[0153] Select training data pairs whose second semantic similarity is greater than the second set value as the fourth data pair, and label them with the second label; or,

[0154] Based on the descending order of the second semantic similarity scores, a second preset number of semantic similarity training data pairs that rank highest are selected as the fourth data pair and labeled with the second tag.

[0155] Furthermore, the semantic similarity between the training data can be calculated using an optimal matching similarity algorithm.

[0156] According to one or more embodiments provided in this disclosure, a training dataset construction apparatus is provided, comprising:

[0157] The basic dataset acquisition module is used to acquire the basic dataset, which includes multiple training data and semantic identifiers corresponding to each training data.

[0158] The first dataset construction module is used to construct a first dataset based on the semantic identifier of the training data. The first dataset includes a first training data pair with a first label and a second training data pair with a second label. The first label indicates that the two training data pairs contained in the first training data pair have the same semantic identifier, and the second label indicates that the two training data pairs contained in the second training data pair have different semantic identifiers.

[0159] The second and third dataset construction modules are used to construct a second dataset and / or a third dataset based on the semantic identifiers of each training data in each of the basic datasets and the text similarity between the data; wherein, the second dataset includes a third training data pair with a first label and a semantic similarity lower than a first preset threshold; the third dataset includes a fourth training data pair with a second label and a semantic similarity higher than a second preset threshold; wherein, the second preset threshold is greater than the first preset threshold;

[0160] A target dataset construction module is used to construct a target dataset based on the second dataset and / or the third dataset, as well as the first dataset, the target dataset being invoked for model training.

[0161] Optionally, the first dataset construction module provided in this disclosure includes:

[0162] The first labeling unit is used to, for any training data, form the first training data pair with the training data by combining a third preset number of training data in the basic dataset that have the same semantic identifier as the training data, and label the training data with the first label.

[0163] The second labeling unit is used to, for any training data, form the second data pair with a fourth preset number of training data in the base dataset that have different semantic identifiers from the training data, and label them with the second label.

[0164] Optionally, the second and third dataset construction modules provided in this disclosure include:

[0165] The first semantic similarity calculation unit is used to calculate the first semantic similarity between training data with the same semantic identifier in the basic dataset;

[0166] The third data pair determination unit is used to select training data pairs whose first semantic similarity is less than the first set value as the third data pair and label them with the first label; or, according to the ascending order of each of the first semantic similarities, select the first preset number of training data pairs that are ranked first as the third data pair and label them with the first label.

[0167] Optionally, the second and third dataset construction modules provided in this disclosure, when constructing the third dataset based on the semantic similarity between training data with different semantic identifiers, can be used for:

[0168] Calculate the second semantic similarity between training data with different semantic labels in the base dataset;

[0169] Select training data pairs with a second semantic similarity greater than a second set value as the fourth data pair, and label them with the second label; or,

[0170] Based on the descending order of the second semantic similarity scores, a second preset number of training data pairs that rank highly are selected as the fourth data pair and labeled with the second tag.

[0171] Optionally, when constructing the second dataset based on the semantic similarity between training data with the same semantic identifier, the semantic similarity between the training data can be calculated by the best matching similarity algorithm.

[0172] According to one or more embodiments provided in this disclosure, an electronic device is provided, comprising:

[0173] One or more processors;

[0174] Memory;

[0175] One or more applications, wherein the applications are stored in memory and configured to be executed by one or more processors, the applications being configured to: execute the training dataset construction method described above.

[0176] According to one or more embodiments provided in this disclosure, a computer-readable medium is provided that stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the training dataset construction method described above.

[0177] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0178] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0179] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for constructing a training dataset, characterized in that, include: Obtain the basic dataset, which includes multiple training data sets and semantic identifiers corresponding to each training data set; A first dataset is constructed based on the semantic identifiers of the training data. The first dataset includes a first training data pair with a first label and a second training data pair with a second label. The first label indicates that the two training data pairs contained in the first training data pair have the same semantic identifier, and the second label indicates that the two training data pairs contained in the second training data pair have different semantic identifiers. Based on the semantic identifiers and text similarity between training data in each of the aforementioned basic datasets, a second dataset and / or a third dataset are constructed; wherein, the second dataset includes a third training data pair with a first label and a semantic similarity lower than a first preset threshold; the third dataset includes a fourth training data pair with a second label and a semantic similarity higher than a second preset threshold; wherein, the second preset threshold is greater than the first preset threshold; A target dataset is constructed based on the second dataset and / or the third dataset, as well as the first dataset, and the target dataset is used for model training.

2. The method according to claim 1, characterized in that, The step of constructing the first dataset based on the semantic identifiers of the training data includes: For any of the training data, a third preset number of training data in the basic dataset that have the same semantic identifier as the training data are respectively combined with the training data to form the first training data pair, and the first label is marked. For any of the training data, a fourth preset number of training data in the base dataset that have different semantic identifiers from the training data are respectively combined with the training data to form a second training data pair, and labeled with the second label.

3. The method according to claim 1, characterized in that, The second dataset is constructed based on the semantic identifiers of each training data point in each of the aforementioned basic datasets and the text similarity between the data points, including: Calculate the first semantic similarity between training data with the same semantic identifier in the basic dataset; Select training data pairs whose semantic similarity is less than a first set threshold as third training data pairs, and label them with the first label; or, Based on the ascending order of each of the first semantic similarities, select the first preset number of semantic similarity training data pairs that rank highest as the third training data pairs and label them with the first label.

4. The method according to claim 1, characterized in that, The third dataset is constructed based on the semantic identifiers of each training data point in each of the aforementioned basic datasets and the textual similarity between the data points, including: Calculate the second semantic similarity between training data with different semantic identifiers in the basic dataset; Select training data pairs whose second semantic similarity is greater than a second set threshold as the fourth training data pair, and label them with the second label; or, Based on the descending order of the second semantic similarity scores, a second preset number of semantic similarity training data pairs that rank highest are selected as the fourth training data pairs and labeled with the second label.

5. The method according to claim 1, characterized in that, The semantic similarity between the training data is calculated using the best matching similarity algorithm.

6. A training dataset construction apparatus, characterized in that, include: The basic dataset acquisition module is used to acquire the basic dataset, which includes multiple training data and semantic identifiers corresponding to each training data. The first dataset construction module is used to construct a first dataset based on the semantic identifier of the training data. The first dataset includes a first training data pair with a first label and a second training data pair with a second label. The first label indicates that the two training data pairs contained in the first training data pair have the same semantic identifier, and the second label indicates that the two training data pairs contained in the second training data pair have different semantic identifiers. The second and third dataset construction modules are used to construct a second dataset and / or a third dataset based on the semantic identifiers of each training data in each of the basic datasets and the text similarity between the data; wherein, the second dataset includes a third training data pair with a first label and a semantic similarity lower than a first preset threshold; the third dataset includes a fourth training data pair with a second label and a semantic similarity higher than a second preset threshold; wherein, the second preset threshold is greater than the first preset threshold; A target dataset construction module is used to construct a target dataset based on the second dataset and / or the third dataset, as well as the first dataset, the target dataset being invoked for model training.

7. The training dataset construction apparatus according to claim 6, characterized in that, The first dataset construction module includes: The first labeling unit is used to, for any training data, form the first training data pair with the training data by combining a third preset number of training data in the basic dataset that have the same semantic identifier as the training data, and label the training data with the first label. The second labeling unit is used to, for any training data, form a second training data pair with the training data by combining a fourth preset number of training data in the base dataset that have different semantic identifiers from the training data, and label them with the second label.

8. The training dataset construction apparatus according to claim 6, characterized in that, The second dataset construction module includes: The first semantic similarity calculation unit is used to calculate the first semantic similarity between training data with the same semantic identifier in the basic dataset; The third training data pair determination unit is used to select training data pairs whose first semantic similarity is less than a first preset threshold as third training data pairs and label them with the first label; or, according to the ascending order of each of the first semantic similarities, select the first preset number of training data pairs that are ranked first as the third training data pairs and label them with the first label.

9. An electronic device, characterized in that, It includes: One or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to: perform the training dataset construction method according to any one of claims 1 to 5.

10. A computer-readable medium, characterized in that, The readable medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the training dataset construction method as described in any one of claims 1 to 5.

Images