Information processing apparatus, information processing method, and computer-readable storage medium

Abstract

An information processing device, an information processing method, and a computer-readable storage medium are provided. The information processing device includes processing circuitry configured to, for each of a plurality of indicators, construct a set of sample units for the indicator from a plurality of labeled minimum labeled sample units relating to the indicator obtained from an original sample set, and for at least a portion of the constructed sets of sample units, extract a minimum labeled sample unit from each set of sample units and generate a labeled training sample based on the extracted minimum labeled units. According to at least one aspect of embodiments of the present disclosure, the sets of sample units are constructed from artificially labeled minimum labeled sample units and labeled training samples are automatically generated based on such sets of sample units, thereby enabling the generation of labeled training samples in a certain degree of automated manner, reducing human involvement.

Description

Technical Field

[0001] This application relates to the field of data processing and / or data analysis, and more specifically, to an information processing apparatus, an information processing method, a computer-readable storage medium, and an information processing apparatus storing a recognition and classification model for text in a specific domain. Background Technology

[0002] With the development of deep learning technology, in recent years, more and more deep learning models and methods have been used to solve tasks in different fields. The powerful feature learning and representation capabilities of deep learning models are acquired through training with a large number of labeled samples.

[0003] In fields such as Natural Language Processing (NLP), it is sometimes necessary to train deep learning models using massive amounts of training samples (e.g., hundreds of thousands to millions) to achieve the performance required for practical applications. Obtaining such a huge number of training samples entirely through manual annotation requires significant time and financial resources. For many specific tasks in various vertical domains, the investment far outweighs the output, posing a significant challenge to practical applications. Therefore, there is a need for a method that can obtain labeled samples more efficiently than purely manual annotation.

[0004] On the other hand, existing deep learning models in natural language processing (NLP) cannot meet the application needs of many vertical domains. Pre-trained models, as an important direction in NLP development, are currently gaining attention in specific applications within vertical domains. Taking classification tasks as an example, a pre-trained classification model can be obtained by first training with generalized text, and then fine-tuned using specific vertical domain text data to obtain a final classification model for that vertical domain. However, due to the difference between the generalized text used for pre-training and the real text data from the vertical domain, the classification performance of the final model is still not ideal. Therefore, for specific tasks in vertical domains, there is a desire to provide classification models with higher accuracy. Summary of the Invention

[0005] A brief overview of this disclosure is given below to provide a basic understanding of certain aspects of it. However, it should be understood that this overview is not an exhaustive summary of this disclosure. It is not intended to identify key or essential parts of this disclosure, nor is it intended to limit the scope of this disclosure. Its purpose is merely to present certain concepts of this disclosure in a simplified form as a prelude to the more detailed description that follows.

[0006] In view of the above problems, at least one aspect of the present disclosure aims to provide an information processing device, an information processing method, and a computer-readable storage medium that constructs a sample unit set based on manually annotated minimum annotation sample units and automatically generates annotated training samples based on such sample unit set, thereby enabling the generation of annotated training samples in a certain degree of automation.

[0007] Furthermore, at least another aspect of this disclosure aims to provide an information processing device that stores a recognition and classification model for domain-specific text. This recognition and classification model improves the final recognition and classification performance of the recognition and classification model by combining a word vector extraction model pre-trained on non-domain-specific text (such as generalized general text) with an untrained context representation model, thereby achieving the benefits of both.

[0008] According to one aspect of this disclosure, an information processing apparatus is provided, comprising a processing circuit configured to: for each of a plurality of indicators, construct a sample unit set for that indicator based on a plurality of labeled minimum labeled sample units related to that indicator obtained from an original sample set; and for at least a portion of the constructed sample unit sets, extract minimum labeled sample units from each sample unit set, and generate labeled training samples based on the extracted minimum labeled units.

[0009] According to another aspect of this disclosure, an information processing apparatus is also provided, comprising a memory storing a recognition and classification model for text specific to a particular domain. The recognition and classification model is used to identify portions of text input related to at least one indicator from the text input specific to the domain, and to classify the identified portions according to the relevant indicator. The recognition and classification model includes: a word vector extraction model pre-trained using at least a first text sample set not specific to the particular domain, for extracting word vectors from the text input; and a context representation model trained using a second text sample set specific to the domain, for outputting the classification result based on the word vectors extracted by the word vector extraction model.

[0010] According to another aspect of this disclosure, an information processing method is also provided, comprising: for each of a plurality of indicators, constructing a sample unit set for the indicator based on a plurality of labeled minimum labeled sample units related to the indicator obtained from an original sample set; and for at least a portion of the constructed sample unit sets, extracting minimum labeled sample units from each sample unit set, and generating labeled training samples based on the extracted minimum labeled units.

[0011] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing executable instructions is also provided, which, when executed by a processor, cause the processor to perform the various functions of the aforementioned information processing method or information processing apparatus.

[0012] In accordance with other aspects of this disclosure, computer program code and computer program products for implementing the methods described above according to this disclosure are also provided.

[0013] According to at least one aspect of the embodiments of this disclosure, a sample unit set is constructed based on the smallest manually annotated sample unit and labeled training samples are automatically generated based on such sample unit set, thereby enabling the generation of labeled training samples in a certain degree of automation, reducing manual intervention, and thus reducing the time and economic costs of obtaining labeled training samples.

[0014] Furthermore, according to at least another aspect of the embodiments of this disclosure, in the recognition and classification model, by combining a word vector extraction model that utilizes text pre-training that is not specific to a particular domain with an untrained context representation model, the former's good feature representation ability regarding word vectors and the latter's good context representation ability in a particular domain are obtained, thereby improving the final recognition and classification performance of the recognition and classification model.

[0015] Other aspects of embodiments of this disclosure are set forth in the following description section, wherein preferred embodiments of the present disclosure are described in detail without limiting them. Attached Figure Description

[0016] The accompanying drawings described herein are for illustrative purposes only and not for all possible implementations, and are not intended to limit the scope of this disclosure. In the drawings:

[0017] Figure 1 This is a block diagram illustrating a configuration example of an information processing device according to a first embodiment of the present disclosure;

[0018] Figure 2 This is a block diagram illustrating a first configuration example of a generation unit in an information processing apparatus according to a first embodiment of the present disclosure;

[0019] Figure 3 This is a block diagram illustrating a second configuration example of a generation unit in an information processing apparatus according to a first embodiment of the present disclosure;

[0020] Figure 4 This is a block diagram illustrating a third configuration example of a generation unit in an information processing apparatus according to a first embodiment of the present disclosure;

[0021] Figure 5 It is shown Figure 4A block diagram of a configuration example of the control subunit in the third configuration example of the generation unit shown;

[0022] Figure 6 This is a block diagram illustrating a fourth configuration example of a generation unit in an information processing apparatus according to a first embodiment of the present disclosure;

[0023] Figure 7 This is a flowchart illustrating a process example of an information processing method according to a first embodiment of the present disclosure;

[0024] Figure 8 This is a block diagram illustrating a configuration example of an information processing device according to a second embodiment of the present disclosure;

[0025] Figure 9 This is an explanatory diagram illustrating an example application scenario of a classification and recognition model stored in the memory of an information processing device applying the second embodiment;

[0026] Figure 10 This is an illustrative diagram showing a performance comparison between the classification and recognition models of Examples 1 and 2 according to this disclosure and the classification and recognition algorithm of Comparative Example 1;

[0027] Figure 11 This is a block diagram of an exemplary structure of a general-purpose personal computer in which methods and / or devices and / or systems according to embodiments of the present disclosure can be implemented.

[0028] While this disclosure is readily subject to various modifications and substitutions, specific embodiments thereof have been shown by way of example in the accompanying drawings and are described in detail herein. However, it should be understood that the description of specific embodiments herein is not intended to limit this disclosure to the specific forms disclosed, but rather, this disclosure is intended to cover all modifications, equivalents, and substitutions falling within the spirit and scope of this disclosure. It should be noted that throughout the drawings, corresponding reference numerals indicate corresponding parts. Detailed Implementation

[0029] Examples of this disclosure will now be described more fully with reference to the accompanying drawings. The following description is merely exemplary and is not intended to limit the disclosure, its application, or its uses.

[0030] Example embodiments are provided so that this disclosure will become exhaustive and will fully convey its scope to those skilled in the art. Numerous specific details, such as examples of particular components, apparatus, and methods, are set forth to provide a detailed understanding of embodiments of this disclosure. It will be apparent to those skilled in the art that the specific details are not required, and that the example embodiments may be implemented in many different forms, none of which should be construed as limiting the scope of this disclosure. In some example embodiments, well-known processes, well-known structures, and well-known techniques are not described in detail.

[0031] The description will proceed in the following order:

[0032] 1. Overview

[0033] 2. Configuration example of the information processing device in the first embodiment

[0034] 2.1 A Configuration Example of an Information Processing Device

[0035] 2.1.1 Preprocessing for obtaining the smallest labeled unit

[0036] 2.1.2 Processing related to building blocks

[0037] 2.1.3 Processing related to generation unit (first configuration example)

[0038] 2.2 Example of preferred configuration of generation unit

[0039] 2.2.2 Example of the second configuration of the generation unit

[0040] 2.2.3 Example of the third configuration of the generation unit

[0041] 2.2.4 Example of the fourth configuration of the generation unit

[0042] 3. Information processing method of the first embodiment

[0043] 4. Configuration example of the information processing device in the second embodiment

[0044] 4.1 A Configuration Example of an Information Processing Device

[0045] 4.2 Sample set used for training models in information processing devices

[0046] 4.3 Example Applications of Information Processing Equipment

[0047] 5. Exemplary structure of a general-purpose personal computer for the methods and / or devices used in the embodiments.

[0048] <1. Overview>

[0049] In fields such as Natural Language Processing (NLP), it is sometimes necessary to train deep learning models using hundreds of thousands to millions of training samples to achieve model performance that meets the requirements of practical applications. For example, deep learning models involving multi-label and multi-task implementations need to perform multiple tasks (e.g., multiple classification tasks) related to multiple metrics. Each sample used to train the model may have more than one label (i.e., several labels related to some or all of the multiple tasks). The parameters of such complex deep learning models can be in the millions, necessitating that the training samples used to train the model also be on a corresponding scale to avoid overfitting.

[0050] Multi-label, multi-task deep learning models have broad application prospects: for example, analyzing user reviews of products on product pages across different dimensions (different metrics or different tasks) to determine the intent or sentiment of user reviews (e.g., positive, negative, or neutral sentiment); analyzing and determining the sentiment of comments on books and movies on social media; and analyzing and determining the sentiment of user feedback received from human customer service. These applications all belong to specific tasks within vertical domains. Obtaining a massive amount of training samples that meet the model's performance requirements entirely based on manual annotation requires high time and economic costs, with the input far exceeding the output, posing a significant challenge to practical applications.

[0051] In view of the above problems, one aspect of this disclosure proposes the concept of a minimum labeled sample unit, and further provides an information processing device, an information processing method, and a computer-readable storage medium, which constructs a sample unit set based on manually labeled minimum labeled sample units and automatically generates labeled training samples based on such sample unit set, thereby enabling the generation of labeled training samples in a certain degree of automation. Therefore, it can reduce manual intervention and lower the time and economic costs of obtaining labeled training samples.

[0052] On the other hand, in natural language processing, existing deep learning models themselves cannot meet the application needs of many vertical fields. One of the current hot topics in specific applications of vertical fields is pre-trained classification models. However, due to the difference between the generalized general text used in pre-training and the real text data from the vertical field later, the final classification model, even after fine-tuning the pre-trained classification model using real text data from the vertical field, still performs unsatisfactorily.

[0053] In view of the above problems, another aspect of this disclosure proposes to combine a model pre-trained using non-domain-specific text (such as generalized general text) with an unpre-trained model, thereby providing an information processing device that stores a recognition and classification model for domain-specific text. This model combines a word vector extraction model pre-trained using non-domain-specific text with an unpre-trained context representation model, thereby obtaining the former's good feature representation ability regarding word vectors and the latter's good context representation ability in the specific domain, thus improving the final recognition and classification performance of the entire model.

[0054] Next, various embodiments based on the above aspects of this disclosure will be described in detail with reference to the accompanying drawings.

[0055] <2. Configuration Example of the Information Processing Device in the First Embodiment>

[0056] The information processing apparatus of the first embodiment of this disclosure relates to generating labeled training samples in a manner that is automated to a certain extent.

[0057] [2.1 A configuration example of an information processing device]

[0058] Figure 1 This is a block diagram illustrating a configuration example of an information processing device according to a first embodiment of the present disclosure.

[0059] like Figure 1 As shown, the information processing device 100 may include a construction unit 110 and a generation unit 120.

[0060] Here, each unit of the information processing device 100 can be included in a processing circuit. It should be noted that the information processing device 100 may include one or more processing circuits. Furthermore, the processing circuit may include various discrete functional units to perform various different functions and / or operations. It should be noted that these functional units can be physical entities or logical entities, and units with different names may be implemented by the same physical entity.

[0061] Construction unit 110 can be configured to construct a sample unit set for each of the multiple metrics, based on multiple labeled minimum labeled sample units related to that metric obtained from the original sample set. Generation unit 120 can be configured to extract minimum labeled sample units from at least a portion of the constructed sample unit sets, and generate labeled training samples based on the extracted minimum labeled units.

[0062] In the context of this disclosure, metrics can be associated with the task to which the labeled training samples are to be applied, and one metric can correspond to one task (or, where appropriate, be equivalent to one task). For example, if the constructed labeled training samples are subsequently applied to a classification task, then one metric can correspond to a classification task. For each of a plurality of metrics, the smallest labeled unit relating to that metric can be labeled with one of a plurality of predetermined labels relating to the task of that metric.

[0063] Preferably, the labeled training samples generated using the information processing device 100 of this embodiment can be training samples to be applied to a classification task. In this case, for each of a plurality of indicators, the smallest labeled unit related to that indicator is labeled with one of a plurality of predetermined labels for the classification task of that indicator. The smallest labeled unit will be further described later when describing the preprocessing for obtaining the smallest labeled unit.

[0064] Using the information processing device 100 of this embodiment, labeled training samples suitable for multi-task (multi-label multi-task) processing can be generated in a degree of automation. Therefore, manual intervention can be reduced, thereby lowering the time and economic costs of obtaining labeled training samples.

[0065] The information processing device 100 of this embodiment can be applied to various types of original samples or original sample sets, including but not limited to original text samples, original image samples, etc. Preferably, the original sample set may include multiple original text samples. In this case, each of the multiple indicators may be related to specific content involved in the text sample, and each minimum annotation unit may, for example, include the minimum set of consecutive sentences describing the corresponding indicator.

[0066] For ease of explanation, the following description will use a text sample as an example, combined with a specific application scenario, to describe the processing performed by the information processing device 100 and its various units, as well as the details of related preprocessing. However, based on this disclosure, those skilled in the art can appropriately apply the information processing device of this embodiment to various scenarios and to other types of raw samples besides text samples.

[0067] First, suppose we want to apply the labeled training samples generated by information processing device 100 to an application scenario of analyzing user reviews and determining intent for electronic products such as headphones: For example, on a product page of an online shopping mall, we want to analyze user feedback on a pair of headphones from the perspective of N indicators (e.g., sound quality, noise cancellation, appearance, cost-effectiveness, Bluetooth performance, etc., where N is a natural number) and predict the sentiment of the descriptions related to these indicators (e.g., one of M sentiments such as positive, negative, or neutral, where M is a natural number). This process is equivalent to identifying the parts of user reviews that involve relevant indicators and classifying their sentiment. This is a typical multi-label, multi-task problem in a specific vertical domain, which can be solved using a deep learning model trained based on labeled training samples.

[0068] Therefore, assuming that hundreds of thousands of user reviews about the headphones have been collected as raw samples, and that these raw samples are used to generate labeled training samples that can be used to train the aforementioned deep learning model in a certain degree of automation.

[0069] (2.1.1 Preprocessing to obtain the smallest labeled unit)

[0070] This disclosure proposes the idea of ​​the minimum annotation unit, which obtains the minimum annotation unit from the original sample set by designing the standard of the minimum annotation unit.

[0071] For example, considering the user reviews of the headphones above, assuming we analyze user feedback on the headphones from the perspective of N=13 indicators (e.g., sound quality, noise cancellation, appearance, cost-effectiveness, Bluetooth performance, etc.), and expect the sentiment of the description of each indicator to be one of M=3 categories (e.g., positive, negative, or neutral), then for a given user review, we expect it to be labeled with tags indicating the indicators involved in the review and the corresponding sentiment category. In other words, we want each user review used in the final labeled training sample to be labeled with a tag corresponding to each indicator involved, with each tag indicating the corresponding indicator and one of the M categories corresponding to that indicator. If a user review involves all N indicators, then after labeling, the user review will have N tags. Optionally, noise category tags can also be defined and labeled separately, as detailed below.

[0072] Various appropriate methods can be used to obtain the final labeled minimum annotation units, which will be used to generate labeled training samples of the above form, through manual preprocessing. Here, the details of the preprocessing for obtaining the minimum annotation units will be described using user reviews of headphones as an example.

[0073] For example, we can first define the scope of application for each indicator. Taking the indicator "sound quality" as an example, its scope of application can be defined as follows: direct evaluation of sound effects, timbre, and sound quality; evaluation of clarity; evaluation of various frequency bands and ranges; and evaluation of resolution. In addition, problems such as harshness, distortion, and abnormal noise can also be classified as sound quality effects, while "no sound" can be excluded from the category of sound quality effects.

[0074] Next, feature keywords or phrases describing each indicator can be defined based on the applicable scope of the indicators defined above, thereby constructing an indicator feature library. Taking the "sound quality" indicator as an example, the feature library composed of feature keywords describing sound quality could include {sound effects, midrange, bass, treble, low frequency, mid frequency, high frequency, sound, range, vocals, LDAC, Hi-Fi, pop, resolution, clarity, timbre, ...}. In a similar manner, an indicator feature library for all N=13 indicators can be constructed.

[0075] In addition, it is necessary to define the labeling rules for each indicator (corresponding to the classification rules). As an example, the labeling (i.e., tag) for relevant samples can include two parts: the first part represents the indicator number, for example, the numbers corresponding to the 13 indicators such as sound quality, noise reduction, appearance, cost-effectiveness, Bluetooth performance, etc., can be the letters "A", "B", "C", "D", "E", etc., in alphabetical order; the second part can represent the sentiment category determined for the indicator, for example, "100" represents a positive (positive, affirmative) description and evaluation of the indicator, "001" represents a negative (negative, negative) description and evaluation of the indicator, and "010" represents a neutral description and evaluation of the indicator. The specific criteria for determining positive, negative, or neutral can be based on the sentiment tendency of the words in the description and the overall sentiment tendency of all relevant words. Alternatively, the labeling method can be designed as a whole, without using the above two-part form, that is, the entire label represents the combination of indicator and category, which will not be elaborated here.

[0076] In this way, the applicable scope and labeling rules for all N=13 indicators can be defined.

[0077] Optionally, the applicable scope and labeling rules for noise samples can be further defined. In this example, noise can be defined as the text portion that interferes with the feature classification of the metric, and can be divided into relevant noise and irrelevant noise. Relevant noise has some similarity to the features of the metric (or involves the description of the metric but has no emotional connotation), while irrelevant noise is completely unrelated to the features of the metric. Taking the "sound quality" metric as an example, an example of its relevant noise could be "The sound is not very loud, but overall it is satisfactory." This text sample does not evaluate the sound quality effect but involves a description of the volume. Taking the "Bluetooth performance" metric as an example, an example of its relevant noise could be "The mobile network connection is sometimes unstable, and music playback is choppy." This sample involves a description of the mobile network rather than Bluetooth performance. An example of irrelevant noise for all 13 metrics could be "Irrelevant. Just got it today, hoping it will bring me some surprises."

[0078] Similar to the labeling of text samples involving actual metrics, the labeling (i.e., tags) of noise samples can also include two parts. The first part indicates the number of the metric related to the noise sample or indicates that the noise sample is unrelated to all metrics. For example, the first part of the label for a noise sample related to one of the metrics such as sound quality, noise reduction, appearance, cost-effectiveness, Bluetooth performance, etc., can be the corresponding metric number, such as a letter from "A", "B", "C", "D", "E", etc., in alphabetical order. The first part of the label for a noise sample unrelated to all 13 metrics can be the number "Z", indicating that it is unrelated to all metrics. The second part of the label for a noise sample can be "000", indicating that the sample is noise. For example, an example of a noise sample related to the sound quality metric, "The sound is not very loud, but overall it is satisfactory," can be labeled "A000", an example of a noise sample related to the Bluetooth metric, "The mobile network connection is sometimes unstable, and music playback is choppy," can be labeled "E000", and an example of a noise sample unrelated to any metric, "Just got it today, hoping it will bring me some surprises," can be labeled "Z000".

[0079] In this way, the range and labeling rules of the relevant noise and irrelevant noise for all N=13 indicators can be defined.

[0080] Based on the applicable scope and annotation rules of all N=13 defined indicators, the smallest annotation unit can be extracted and labeled. Here, the smallest annotation unit is defined, for example, as the smallest continuous sentence combination that satisfies the description of the indicator, and the specific extraction method of the smallest annotation unit can be, for example, to extract the smallest continuous sentence combination containing feature keywords from the indicator feature library.

[0081] Taking the "sound quality" index as an example, assuming the original sample is "The sound is clear, the effect is good, the bass is powerful, and the treble is bright. The appearance is quite cool, but the price is a bit high. I bought it for my boyfriend, and I hope he likes it," then the smallest labeled unit of the "sound quality" index extracted from it using the feature keywords (bass, treble, sound) in the feature library of sound quality indexes can be "The sound is clear, the effect is good, the bass is powerful, and the treble is bright." The labeling result for this is "A100," which represents a positive description and evaluation of the sound quality.

[0082] In this way, the required number of minimum labeled units can be extracted and labeled for all N=13 indicators. As an example, when hundreds of thousands of training samples are needed, tens of thousands (e.g., 10,000 to 30,000) of the original samples can be extracted from the original samples to label the minimum labeled units, and the labeled minimum labeled units can be obtained for subsequent processing.

[0083] In this process, to improve the efficiency of manual annotation, a pre-designed user interface (UI) can be used to assist in the manual annotation process. For example, the UI can combine with a previously built indicator feature library to highlight relevant feature keywords, thereby improving the efficiency of manual annotation through an attention mechanism.

[0084] Alternatively, given that the scope of application and labeling rules for the noise samples are defined, a predetermined number of noise texts can be extracted from the original sample set, and preferably may include irrelevant noise and / or relevant noise of each indicator for use in subsequent processing.

[0085] The above example, using user reviews of headphones, describes the details of preprocessing for extracting and labeling the smallest annotation units based on the applicable scope and annotation rules of the metrics. Although the applicable scope and annotation rules of the metrics are used to extract and label the smallest annotation units in the above description, they can also be used for direct labeling of the original samples. In other words, since the smallest annotation units will ultimately be used to generate labeled training samples with a similar format to the labeled original samples, the applicable scope and annotation rules of the metrics, which are also applicable to the labeling of the original samples, can be used in the preprocessing for extracting and labeling the smallest annotation units.

[0086] Based on the above description, those skilled in the art will understand that, for different original samples and / or application scenarios, various appropriate methods can be used to define the applicable scope and annotation rules of suitable indicators, and correspondingly implement preprocessing to extract and annotate the smallest annotation unit, as long as the smallest annotation unit that meets the requirements can be obtained. For example, taking the original text sample as an example, it is sufficient to extract the smallest continuous sentence combination that satisfies the description of the indicator from the original text sample (for example, this can be achieved by extracting the smallest continuous sentence combination containing feature keywords from the indicator feature library).

[0087] (2.1.2 Processing related to building blocks)

[0088] Next, we will describe the processing related to building blocks.

[0089] As described above, the construction unit 110 can be configured to construct a sample unit set for each of the multiple metrics, based on multiple labeled minimum labeled sample units related to that metric obtained from the original sample set.

[0090] For example, considering the user reviews of the headphones above, construction unit 110 can be configured to construct a sample unit set for each of the N=13 metrics, based on the smallest labeled unit related to that metric obtained from the original samples through preprocessing. For example, construction unit 110 can construct 13 sample unit sets Set1, Set2, Set3, ... Set13 for a total of 13 metrics such as sound quality, noise cancellation, appearance, cost-effectiveness, and Bluetooth performance.

[0091] Preferably, each smallest labeled unit in the sample unit set constructed by construction unit 110 involves only one indicator. For example, the smallest labeled unit in the sample unit set constructed for sound quality involves only sound quality and is labeled with only one of the sound quality-related tags "A100", "A001", and "A010". Such a configuration will facilitate subsequent processing. Alternatively, it is possible to appropriately design and implement a case where each smallest labeled unit in the sample unit set involves more than one indicator, which will not be elaborated here.

[0092] (2.1.3 Processing related to the generation unit (first configuration example))

[0093] Next, we will refer to Figure 2 The first configuration example of the generation unit shown is used to describe the processing related to the generation unit.

[0094] like Figure 2As shown, the generation unit 120 of the first configuration example may include an extraction subunit 1210 and a generation subunit 1220. For example, considering the example of user reviews of the headphones above, for at least a portion of the 13 sample unit sets Set1, Set2, Set3, ... Set13 constructed, the extraction subunit 1210 may be configured to extract the minimum labeled sample unit from each sample unit set, and the generation subunit 1220 may be configured to generate labeled training samples based on the extracted minimum labeled sample unit.

[0095] Here, "at least a portion of the sample unit set" can be specified, for example, based on the labeled training samples to be generated. For instance, assuming that there are insufficient labeled training samples for certain metrics, the corresponding sample unit set can be specified based on such metrics to generate labeled training samples involving the relevant metrics.

[0096] Preferably, the extraction subunit 1210 can randomly extract a minimum labeled sample unit from each sample unit set in the at least part of the sample unit set. Here, assuming that each minimum labeled unit in the sample unit set constructed by the construction unit 110 involves only one indicator, then the extraction subunit 1210 randomly extracting a minimum labeled sample unit from a given sample unit set can mean that the extracted minimum labeled unit only involves the indicator of the given sample unit set and has only one label related to that indicator. Such an extraction method will facilitate the direct determination of the number and content of indicators involved in the extracted minimum labeled sample unit based on the operation process of the extraction subunit 1210.

[0097] The "at least a portion of the sample unit set" involved in the processing of the generation unit 120 can be one or more sample unit sets. When the sample unit set involved in the processing of the generation unit 120 is a sample unit set, the extraction sub-unit 1210 can randomly extract a minimum labeled sample unit from the sample unit set, and the generation sub-unit 1220 can use the minimum labeled sample unit as the generated labeled training sample.

[0098] Furthermore, when the processing of the generation unit 120 involves more than one set of sample units, such as three sets of sample units, the extraction sub-unit 1210 can randomly extract a minimum labeled sample unit from each of the three sets of sample units, and the generation sub-unit 1220 can generate labeled training samples based on the three extracted minimum labeled units.

[0099] In a preferred embodiment, the generation subunit 1220 can be configured to generate labeled training samples by randomly combining the smallest labeled units extracted by the extraction subunit 1210. The labeled training samples generated by the generation subunit can be, for example, paragraph-level text samples.

[0100] Reference above Figures 1 to 2 A first configuration example of an information processing apparatus according to an embodiment of the present disclosure is described. Using the information processing apparatus of this embodiment, after obtaining the smallest annotation unit, it is possible to automatically generate labeled training samples suitable for multi-task (multi-label multi-task) processing. Therefore, the information processing apparatus of this embodiment helps to reduce manual intervention and lower the time and economic costs of obtaining labeled training samples.

[0101] [2.2 Example of Preferred Configuration of Generation Unit]

[0102] Next, we will describe preferred configuration examples of the generation unit, namely the second to fourth configuration examples of the generation unit.

[0103] (2.2.2 Example of the second configuration of the generation unit)

[0104] First, refer to Figure 3 The second configuration example for the generation unit is described.

[0105] Figure 3 This is a block diagram illustrating a second configuration example of a generation unit in an information processing apparatus according to a first embodiment of the present disclosure.

[0106] like Figure 3 As shown, the generation unit 320 of the second configuration example may include an extraction subunit 3210 and a generation subunit 3220, which respectively correspond to Figure 2 The extraction subunit 1210 and the generation subunit 1220 of the generation unit 120 shown. Figure 3 The generation unit 320 shown is Figure 2 The difference in the generation unit 120 shown is that it additionally includes an optional determination subunit 3230, and the following description will focus on this difference.

[0107] The determining subunit 3230 can be configured to randomly determine at least a portion of the sample unit sets from the constructed plurality of sample unit sets. Preferably, when the generating unit 320 includes the determining subunit 3230, each subunit can be configured to repeatedly perform its respective process to obtain a training sample set including a plurality of labeled training samples. That is, the determining subunit 3230 can repeatedly and randomly determine at least a portion of the sample unit sets, the extraction subunit 3210 can repeatedly extract (e.g., randomly extract) the smallest labeled sample unit from each determined sample unit set, and the generating subunit 3220 can repeatedly generate labeled training samples based on the extracted smallest labeled sample unit. Accordingly, each repeated process of each subunit can jointly generate one labeled training sample, and multiple repeated processes can generate multiple labeled training samples, thereby obtaining a training sample set.

[0108] Using the above methods, a training sample set containing a large number of labeled training samples can be automatically constructed based on the sample unit set built by the building unit, thereby greatly reducing the degree of manual intervention and improving annotation efficiency. Theoretically, the number of labels and the workload of manual annotation are positively exponentially related; therefore, the more indicators involved and / or the more categories corresponding to each indicator, resulting in a larger number of labels, the more obvious the advantage of using the generation unit in this configuration example to automatically generate the training sample set, and the greater the time and cost savings.

[0109] Furthermore, when the required number of training samples exceeds the total number of original samples in the original sample set, the generation unit in this configuration example also provides the possibility of generating training samples that meet the requirements. For example, for the constructed 13 sample unit sets Set1, Set2, Set3, ... Set13, assuming that the minimum number of labeled samples in each sample unit set is C1, C2, C3, ... C13 respectively, then by using the repeated processing of the generation unit in this configuration example, the maximum number of labeled training samples that can be generated theoretically is Nmax = C1 * C2 * C3 ... * C13, which can be much larger than the total number of original samples.

[0110] (2.2.3 Example of the third configuration of the generation unit)

[0111] Next, we will refer to Figure 4 and Figure 5 A third configuration example describing the generation unit.

[0112] Figure 4 This is a block diagram illustrating a third configuration example of a generation unit in an information processing apparatus according to a first embodiment of the present disclosure.

[0113] like Figure 4As shown, the generation unit 420 of the third configuration example may include an extraction subunit 4210, a generation subunit 4220, and a determination subunit 4230, which respectively correspond to Figure 3 The generation unit 320 shown includes the extraction subunit 3210, the generation subunit 3220, and the determination subunit 3230. Figure 4 The generation unit 420 shown is Figure 3 The difference in the generation unit 320 shown is that it also includes an optional control subunit 4240, and the following description will focus on this difference.

[0114] The control subunit 4240 can be configured to: determine the expected distribution of the number of indicators involved in each labeled training sample in the obtained training sample set based at least in part on the original distribution of the number of indicators involved in multiple original samples in the original sample set, and make the actual distribution of the number of indicators involved in each labeled training sample conform to the expected distribution by controlling the repeated execution of the extraction subunit 4210, the generation subunit 4220, and the determination subunit 4230 respectively.

[0115] Figure 5 It shows Figure 4 The diagram shows a configuration example of the control subunit 4240. For example... Figure 5 As shown, the control subunit 4240 may include a distribution determination module 4241 and a distribution control module 4242. The distribution determination module 4241 may be configured to determine the desired distribution based at least in part on the original distribution of the number of metrics involved in multiple original samples in the original sample set. The distribution determination module 4241 determines the desired distribution based at least in part on the original distribution, such that the desired distribution can reflect or simulate the original distribution to a certain extent. This will enable the model trained using a training sample set with such a desired distribution to have good performance when applied to real samples with the original distribution.

[0116] As an example, the distribution determination module 4241 can be configured such that the determined desired distribution is based at least in part on a normal distribution whose mean and variance are based on the mean and variance of the number of indicators involved in the multiple original samples in the original sample set.

[0117] Here, it is assumed that, as preprocessing, a portion of the original samples (e.g., 10-20% of the original samples) have been manually labeled, and the original distribution of the number of indicators involved in the sampled original samples has been statistically determined, i.e., the probability of occurrence of samples involving each indicator in the original samples. For example, the number of original samples involving k indicators, count(k), k = 0, 1, 2, 3, ..., 13, can be counted, and the probability of occurrence of samples involving k indicators in the original samples, P(k), can be calculated according to the following formula (1):

[0118]

[0119] The distribution determination module 4241 can determine the required normal distribution based on the original distribution P(k) of the number of indicators involved in the original sample obtained, for example, in the manner described above. For example, the distribution determination module 4241 can calculate the mean μ of the number of indicators involved in the original sample according to the original distribution P(k) according to the following formula (2), and then calculate the standard deviation (variance) σ of the number of indicators involved in the original sample according to formula (3):

[0120]

[0121]

[0122] After obtaining the mean μ and variance σ in this way, the distribution determination module 4241 can determine the normal distribution according to the following formula (4):

[0123]

[0124] Where f(x) represents the probability of occurrence of a sample with x numbers of indicators involved.

[0125] In one example, the distribution determination module 4241 can directly determine the normal distribution shown in formula (4) as the expected distribution of the number of indicators involved in each labeled training sample in the obtained training sample set.

[0126] Using the above-mentioned normal distribution as the expected distribution helps to avoid potential imbalances or disequilibrium issues in the original samples. The original samples may exhibit imbalances between different categories. For example, there might be too many samples of one category, leading to overfitting, or too few samples of another category, leading to underfitting. Both of these situations will affect the final accuracy of the model trained using the original samples. The normal distribution is a relatively balanced distribution. The distribution determination module 4241, for example, using the above formula (4) based on the original distribution, can adjust the imbalance of the original samples to a certain extent (equivalent to smoothing the distribution of the samples), thereby effectively avoiding overfitting and underfitting problems.

[0127] In a preferred embodiment, the distribution determination module 4241 may be further configured to: determine a desired distribution by adjusting the normal distribution using the original distribution, such that in the desired distribution, the probability of occurrence of the number of indicators involved in each labeled training sample is equal to the weighted average of the first probability of occurrence determined according to the normal distribution and the second probability of occurrence determined according to the original distribution.

[0128] For example, the distribution determination module 4241 can calculate the probability f1(k) of the number of indicators involved in each labeled training sample as the probability of the expected distribution according to the following formula (5):

[0129] f1(k)=(f(k)+αP(k)) / 2…Formula (5)

[0130] Where f(k) represents the first probability of occurrence determined by the normal distribution by setting x in formula (4) to k = 0, 1, 2, 3, ..., 13, and P(k) represents the second probability of occurrence determined by the original distribution. Here, a weighting coefficient α, for example, between 0 and 1, is set for the second probability of occurrence to appropriately determine the probability of the final expected distribution.

[0131] In formula (5), the original distribution is further used to adjust the normal distribution. In practice, for example, when the difference between the normal distribution determined by formula (4) and the original distribution is too large, the weighted average of the original distribution and the normal distribution can be used as the expected distribution, making the expected distribution closer to the original distribution than the normal distribution. Samples with such an expected distribution not only adjust the potential imbalance problem of the original samples, but also avoid excessive differences from the original distribution.

[0132] In practical applications, for example, the calculated probability f1(k) of the expected distribution can be further normalized according to the following formula (6) to calculate the final probability f2(k):

[0133]

[0134] Once the distribution determination module 4241 determines the expected distribution, i.e., the probability of occurrence of the number of indicators involved in each labeled training sample, as described above, the distribution control module 4242 can determine the number of labeled training samples involving that number of indicators to be generated based on the probability of occurrence of the number of each indicator and the total number of labeled training samples to be generated. By controlling the repeated execution of the extraction subunit 4210, the generation subunit 4220, and the determination subunit 4230, the required labeled training samples are generated according to the determined number, so that the actual distribution of the number of indicators involved in each labeled training sample conforms to the expected distribution.

[0135] For example, consider the following example: 13 sample unit sets Set1, Set2, Set3, ... Set13 are pre-constructed for N = 13 indicators. The minimum number of labeled samples in each sample unit set is C1, C2, C3, ... C13, where Ci is a natural number between 1000 and 2000 (i = 1, 2, ... 13, representing the sample unit set number), and the minimum labeled sample in each sample unit set involves only one indicator corresponding to that sample unit (i.e., it only has one label related to that one indicator). It is desired to generate a training sample set with a sample number Nsample = 3,000,000 based on such sample unit sets, and the distribution of the number of indicators involved in each labeled training sample conforms to the expected distribution determined, for example, according to the above formula (6), that is, the probability of the occurrence of a labeled training sample involving k indicators is f2(k), where k = 0, 1, 2, 3, ..., 13.

[0136] To this end, the distributed control module first determines the number of training samples involving k metrics:

[0137] Nsample k =Nsample*f2(k)…Formula (7)

[0138] Next, for k = 1, 2, ... 13, according to the corresponding number Nsample mentioned above. k Generate labeled training samples.

[0139] Taking k=1 as an example, the distribution control module 4242 can control each sub-unit to repeatedly perform the following process Nsample1 times: control the determined sub-unit 4230 according to the implementation combination formula The first random function randomly determines a sample unit set Seti from the 13 sample unit sets; the extraction subunit 4210 is controlled according to the implementation combination formula. The second random function randomly extracts a minimum labeled unit from the Ci minimum labeled units of the determined sample unit set Seti; the control generation sub-unit 4220 directly uses the extracted minimum labeled unit as the generated labeled training sample. In this way, each labeled training sample in the generated Nsample1 labeled training samples involves only k=1 indicators.

[0140] Taking k=2 as an example, the distribution control module 4242 can control each sub-unit to repeat the following process Nsample2 times: control the determined sub-unit 4230 according to the implementation combination formula The first random function randomly determines two sample unit sets, Seti and Setj, from the 13 sample unit sets (i and j are natural numbers between 1 and 13 and are mutually exclusive, representing the numbers of the two different sample unit sets); the control extraction subunit 4210 implements the combination formula respectively. and Two second random functions randomly extract one minimum labeled unit from the Ci minimum labeled units of the determined sample unit set Seti, and randomly extract one minimum labeled unit from the Cj minimum labeled units of the determined sample unit set Setj; control the generation of sub-units 4220 according to the implementation arrangement formula. The third random function generates labeled training samples by randomly combining (i.e., randomly permuting) the two smallest labeled units extracted. Each of the Nsample2 labeled training samples generated in this way involves k = 2 indicators.

[0141] In this way, for k = 1, 2, ... 13, the distribution control module 4242 can control each sub-unit 4210 to 4230 to repeatedly perform N samples. k Secondary correlation processing to generate Nsamples k There are k labeled training samples, each involving k metrics. All these labeled training samples together constitute the generated training sample set, SampleSet.

[0142] In the examples above, each unit in each unit set is associated with only a single metric, thereby optimizing or simplifying control over the number of metrics involved in the generated labeled training samples. That is, control is achieved solely by, for example, based on the implementation of a combination formula. The number of sample units k randomly selected by the first random function each time can control the number of indicators k involved in the final generated labeled training samples.

[0143] However, the embodiments disclosed herein are not limited thereto. Based on the preferred examples above, different configurations of the sample unit sets can be adopted, and the distribution control module 4242 can perform corresponding appropriate control. For example, the smallest labeled unit in each sample unit set may be associated with more than one metric; for example, a certain smallest labeled unit may involve two or more metrics simultaneously. In this case, the number of metrics involved in the finally generated labeled training samples can be controlled through comprehensive control (e.g., each time a labeled training sample is generated, its label is read to determine the number of metrics involved in the sample), and the distribution of the number of metrics involved in each labeled training sample in the training sample set is controlled accordingly, which will not be elaborated further here.

[0144] (2.2.4 Example of the fourth configuration of the generation unit)

[0145] Next, we will refer to Figure 6 The fourth configuration example describes the generation unit.

[0146] Figure 6 This is a block diagram illustrating a fourth configuration example of a generation unit in an information processing apparatus according to a first embodiment of the present disclosure.

[0147] like Figure 6 As shown, the generation unit 620 of the fourth configuration example may include an extraction subunit 6210, a generation subunit 6220, a determination subunit 6230, and a control subunit 6240, which respectively correspond to Figure 4 The generation unit 420 shown includes the extraction subunit 4210, the generation subunit 4220, the determination subunit 4230, and the control subunit 4240. Figure 6 The generation unit 620 shown is Figure 4 The difference in the generation unit 420 shown is that it additionally includes an optional noise subunit 6250, and the following description will focus on this difference.

[0148] The noise subunit 6250 can be configured to include a predetermined proportion of noise samples in the training sample set. For example, under the control of the control subunit 6240, the extraction subunit 6210, generation subunit 6220, and determination subunit 6230 follow the above-mentioned reference. Figures 4 to 5 After generating a training sample set SampleSet containing labeled training samples whose number of involved indicators conforms to the desired distribution, the noise subunit 6250 can add a predetermined proportion of noise samples to the training sample set.

[0149] Here, noise samples can be, for example, noise samples in the original samples that are pre-annotated according to the annotation rules associated with the noise samples, as described above in the (2.1.1 Preprocessing for obtaining the smallest annotation unit) section, and can include, for example, one or both of relevant noise and irrelevant noise.

[0150] In a preferred embodiment, noise samples can be considered as those original samples in which the number of indicators involved is 0. In this case, the distribution of noise samples in the final training sample set can also conform to the expected distribution described above in the (3rd configuration example of the generation unit) section. For example, the required number of noise samples Nsample0 = Nsample * f2(0) can be determined according to the total number of samples Nsample in the final training sample set and the probability f2(0) of the occurrence of samples involving 0 indicators in the expected distribution, as in Equation (7), and Nsample0 noise samples are put into the training sample set SampleSet.

[0151] Alternatively, a predetermined proportion of noise samples, determined in other ways, can be added to the training sample set. For example, assuming that it is desired that the final noise sample set includes a predetermined proportion of 30% noise samples, the number of noise samples to be added can be determined to be Nnoise = Nsample * 30%. The noise subunit 6240 can add Nnoise noise samples obtained from the original samples to the SampleSet, thereby obtaining a training sample set that includes 30% noise samples.

[0152] In this way, a training sample set including noisy samples can be obtained, which is beneficial to improving the accuracy of subsequent training using the training sample set.

[0153] The above describes a configuration example of the information processing apparatus according to a first embodiment of the present disclosure. As described above, using the information processing apparatus of the first embodiment, after obtaining the smallest annotation unit, it is possible to automatically generate labeled training samples suitable for multi-task (multi-label multi-task) processing. Therefore, the information processing apparatus of this embodiment helps to reduce manual intervention and lower the time and economic costs of obtaining labeled training samples.

[0154] <3. Information Processing Method of the First Embodiment>

[0155] Corresponding to the information processing device of the first embodiment described above, this disclosure provides an information processing method according to the first embodiment.

[0156] Figure 7 This is a flowchart illustrating a process example of an information processing method according to a first embodiment of the present disclosure.

[0157] like Figure 7 As shown, in step S701, for each of the multiple indicators, a sample unit set for that indicator is constructed based on multiple labeled minimum labeled sample units related to that indicator obtained from the original sample set. Here, before step S701, multiple labeled minimum labeled sample units related to each indicator have been obtained from the original sample set in advance through the preprocessing described in section (2.1.1 Preprocessing for Obtaining Minimum Labeled Units).

[0158] Next, in step S702, for at least a portion of the sample unit sets in the constructed multiple sample unit sets, the minimum labeled sample unit is extracted from each sample unit set, and labeled training samples are generated based on the extracted minimum labeled unit.

[0159] Preferably, the labeled training samples generated using the information processing method of this embodiment can be training samples to be applied to a classification task. In this case, for each of the multiple indicators, the smallest labeled unit related to that indicator is labeled with one of a plurality of predetermined labels for the classification task of that indicator.

[0160] The information processing method of this embodiment can be applied to various types of original samples or original sample sets, including but not limited to original text samples, original image samples, etc. Preferably, the original sample set may include multiple original text samples. In this case, each of the multiple indicators may be related to specific content involved in the text sample, and each minimum annotation unit may, for example, include the minimum set of consecutive sentences describing the corresponding indicator.

[0161] Preferably, in step S702, labeled training samples can be generated by randomly combining the extracted smallest labeled units.

[0162] Preferably, step S702 may further include the following optional process: randomly determining at least a portion of the sample unit sets among the constructed plurality of sample unit sets.

[0163] In this case, in step S702, the processes of randomly determining at least a portion of the sample unit set, extracting the smallest labeled sample unit, and generating labeled training samples can be repeatedly performed to obtain a training sample set including multiple labeled training samples.

[0164] Preferably, step S702 may further include the following optional processing: determining the desired distribution based at least in part on the original distribution of the number of indicators involved in multiple original samples in the original sample set, and controlling the repeatedly executed processing so that the distribution of the number of indicators involved in each labeled training sample in the obtained training sample set conforms to the desired distribution.

[0165] As an example, the expected distribution may be based at least in part on a normal distribution whose mean and variance are based on the mean and variance of the number of indicators involved in multiple original samples in the original sample set.

[0166] Preferably, the desired distribution can be determined by adjusting the normal distribution using the original distribution, such that in the desired distribution, the probability of occurrence of the number of indicators involved in each labeled training sample is equal to the weighted average of a first occurrence probability determined according to the normal distribution and a second occurrence probability determined according to the original distribution.

[0167] Optionally, step S702 may also include the following optional process: including a predetermined proportion of noise samples in the training sample set.

[0168] According to embodiments of this disclosure, the entity performing the above method may be the information processing device 100 according to the first embodiment of this disclosure. Therefore, all embodiments of the information processing device 100 and its various constituent units (and sub-units and modules therein) described above are applicable here.

[0169] <4. Configuration Example of the Information Processing Device in the Second Embodiment>

[0170] The information processing apparatus of the second embodiment of this disclosure relates to the recognition and classification of text for a specific domain.

[0171] [4.1 A configuration example of an information processing device]

[0172] Figure 8 This is a block diagram illustrating a configuration example of an information processing device according to a second embodiment of the present disclosure.

[0173] like Figure 8 As shown, the information processing device 800 may include a memory 810. The memory 810 may store a domain-specific text recognition and classification model, which is used to identify portions of text input related to at least one indicator from the domain-specific text input and classify the identified portions according to the relevant indicator. The recognition and classification model may include: a word vector extraction model pre-trained using at least a first text sample set not specific to the domain, for extracting word vectors from the text input; and a context representation model trained using a second text sample set specific to the domain, for outputting the classification result based on the word vectors extracted by the word vector extraction model.

[0174] Here, the text processed by the recognition and classification model stored in memory 810 can include various text samples from various vertical fields, such as user reviews of products on product pages, comments on books and videos on social media, user feedback to customer service on message boards, and so on. Each indicator identified by the recognition and classification model from the text sample is related to the specific content involved in the text. For example, it can include different dimensions for analyzing the above text samples, and the classification for each indicator can be, for example, the intent or sentiment of the user comment or feedback determined by that indicator (e.g., one of the sentiments such as affirmation, negation, or neutrality).

[0175] For text recognition and classification tasks in vertical domains such as those mentioned above, the recognition and classification model stored in memory 810 combines a word vector extraction model pre-trained using a text sample set that is not specific to a particular domain (such as generalized general text) with a context representation model that is not pre-trained (which is trained only using a text sample set specific to a particular domain). This results in the former model having good feature representation ability for word vectors and the latter model having good context representation ability in a particular domain, thus improving the final recognition and classification performance and achieving higher classification accuracy.

[0176] In the recognition and classification model stored in memory 810, the word vector extraction model can extract word vectors from the input text sample (e.g., paragraph-level text sample), and preferably extracts dynamic word vectors. Furthermore, the context representation model can receive the word vectors output by the word vector extraction model as input and output recognition and classification results. The output recognition and classification results can be, for example, a label distribution for the input text, which can be considered as a sequence of labels representing the true results (i.e., the probability distribution corresponding to the label sequence), where each label can include two parts: the first part represents the recognized indicator, and the second part represents the category determined with respect to that indicator.

[0177] Both the word vector extraction model and the context representation model can be implemented using neural network models. For example, the word vector extraction model can be a word embedding model based on a neural network with hidden layers, such as the word2vec model. Preferably, the word vector extraction model can be implemented using a contextualized word embeddings (CWE) model. The context representation model can be, for example, a recurrent neural network model, such as a Long Short-Term Memory (LSTM) model.

[0178] Preferably, one of the word vector extraction model and the context representation model can be based on a recurrent neural network (RNN), and the other model can be based on an attention mechanism.

[0179] For example, word vector extraction models can be based on attention mechanisms, and can be implemented, for instance, through attention-based models within CWE models (such as Bidirectional Encoder Representations from Transformers (BERT) models). When computing a word vector, an attention-based CWE model performs a weighted sum of the word vector and its context word vectors (with weights corresponding to the parameters in the CWE model) and optionally normalizes it. This is equivalent to enhancing the word vector using contextual information, thus introducing an attention mechanism and increasing the semantic representation capability of the word vector.

[0180] Furthermore, the context representation model can be based on a recurrent neural network, and can be implemented, for example, through a bidirectional long short-term memory (Bi-LSTM) model, LSTM, or other similar models. In one example, an RNN-based context representation model could include multiple Bi-LSTM layers, multiple LSTM layers, and a fully connected layer (FC). The number of layers and their parameters can be set appropriately.

[0181] As alternatives, word vector extraction models can be based on recurrent neural networks (RNNs), while context representation models can be based on attention mechanisms. Examples of RNN-based word vector extraction models can include language model-based embeddings from language models (ELMO) models implemented using Bi-LSTM. Examples of attention mechanism-based context representation models can include neural network models based on self-attention mechanisms, implemented using transformer structures with encoders and decoders.

[0182] The complementary structure of the word vector extraction model and the context representation model, which employ attention mechanisms and RNN models respectively, is beneficial for further improving the performance of the entire classification and recognition model.

[0183] Based on the model architecture and the sample sets used for training (a first text sample set that is not specific to any particular domain and is only used to pre-train the word vector extraction model, and a second text sample set that is specific to any particular domain and can be used to train the entire recognition and classification model), those skilled in the art can obtain the optimized parameters of the recognition and classification model through an appropriate training process.

[0184] For example, a word vector extraction model, such as the CWE model, can be trained using a first set of text samples that is not specific to a particular domain, through various existing methods to obtain a pre-trained word vector extraction model.

[0185] Next, a loss function can be constructed based on the correct recognition and classification probability (or simply the correct classification probability) of the entire recognition and classification model. For example, the loss function can represent the difference between the model's output and the correct recognition and classification result. The goal is to obtain the minimum value of the loss function. The entire recognition and classification model can be trained end-to-end using a second text sample set from a specific domain, thereby obtaining the optimal values ​​of each parameter in the entire model.

[0186] Given the established model architecture and the above training methods, those skilled in the art can implement the specific training process using various existing methods, which will not be described in detail here.

[0187] [4.2 Sample set used for training models in information processing devices]

[0188] As described above, in order to train the recognition and classification model stored in the memory 810 of the information processing device 800, a first text sample set that is not specific to a particular domain and a second text sample set that is specific to a particular domain are required.

[0189] On the one hand, a first set of text samples that is not specific to a particular domain can be obtained through generalized texts, which can be obtained relatively easily through various existing means.

[0190] On the other hand, the second text sample set for a specific domain involves text within a vertical domain. For such vertical domain text, it may be difficult to obtain a sufficient number of labeled texts as training data. Therefore, in a preferred embodiment, labeled training samples (or even the training sample set) obtained by the information processing device according to the first embodiment can be used as samples in the second text sample set applied to train the model in this embodiment (or even used as the second text sample set), thereby solving the problem of difficulty in obtaining labeled samples within a vertical domain.

[0191] For ease of explanation, the following description, in conjunction with a specific application scenario involving user reviews of headphones, will detail a preferred embodiment of using the labeled training samples (or even the training sample set) obtained by the information processing device of the first embodiment as samples in the second text sample set (or even the second text sample set used for training the recognition and classification model). Based on this disclosure, those skilled in the art can appropriately apply this preferred embodiment to various scenarios, which will not be elaborated upon here.

[0192] In this example application scenario, consider the example of user reviews of the headphones described in the first embodiment: The classification and recognition model needs to analyze user feedback on the headphones from the perspective of N=13 indicators (e.g., sound quality, noise cancellation, appearance, cost-effectiveness, Bluetooth performance, etc.). This requires identifying the portions of the user review that relate to these 13 indicators and classifying the sentiment of the identified portions regarding the relevant indicators into one of M=3 categories (e.g., positive, negative, or neutral). Optionally, the classification and recognition model can also identify noise categories in the input text.

[0193] In this preferred embodiment, the above references can be used. Figure 1 The information processing apparatus 100 of the first embodiment described herein generates at least one labeled training sample in a second text sample set through processing by the construction unit 110 and the generation unit 120 of the apparatus. In other words, at least one labeled training sample in the second text sample set may be obtained in the following manner: for each of a plurality of indicators, a sample unit set for that indicator is constructed based on a plurality of labeled minimum labeled sample units related to that indicator obtained from an original text sample set in a specific domain; for at least a portion of the constructed sample unit sets, minimum labeled sample units are extracted from each sample unit set, and labeled training samples are generated based on the extracted minimum labeled units.

[0194] Here, the smallest labeled sample unit of the index is obtained in advance, for example, through the preprocessing described in (2.1.1 Preprocessing for obtaining the smallest labeled unit) in the description of the information processing device of the first embodiment above.

[0195] Preferably, for each of the plurality of indicators, the smallest annotation unit relating to that indicator can be labeled with one of a plurality of predetermined labels relating to the task of that indicator. Furthermore, preferably, each of the plurality of indicators is associated with specific content addressed in the text of the particular domain, and each smallest annotation unit comprises the smallest set of consecutive sentences describing the corresponding indicator.

[0196] For example, taking the "sound quality" metric as an example, the smallest unit of annotation related to this metric could be "clear sound, good effect, powerful bass, bright treble," which could be labeled "A100." The first part "A" of the label "A100" represents the number of the metric involved in this smallest unit of annotation (i.e., one of the numbers "A," "B," "C," "D," "E," etc., corresponding to sound quality, noise cancellation, appearance, cost-effectiveness, Bluetooth performance, etc.). The second part "100" of the label "A100" represents the category of this smallest unit of annotation related to the metric involved (i.e., one of "100" indicating a positive description and evaluation of the metric, "001" indicating a negative description and evaluation of the metric, or "010" indicating a neutral description and evaluation of the metric).

[0197] Based on the labeled minimum labeled sample units obtained from the original samples as described above, for example, 13 sample unit sets Set1, Set2, Set3, ... Set13 were constructed for N=13 indicators (such as sound quality, noise reduction, appearance, cost performance, Bluetooth performance, etc.).

[0198] As an example, at least one labeled training sample generated may be generated by randomly combining the smallest labeled units extracted from each of the sample unit sets in at least a portion of the 13 sample unit sets mentioned above.

[0199] In one example, the "at least a subset of sample units" used to extract the minimum labeled samples can be specified, for example, based on the labeled training samples to be generated. For instance, assuming that there are insufficient labeled training samples for certain metrics, a corresponding set of sample units can be specified based on such metrics to generate labeled training samples involving the relevant metrics.

[0200] Alternatively, the "at least a subset of sample units" used to extract the minimum labeled sample can be randomly determined from among the constructed subsets of sample units. For example, the reference in the information processing device of the first embodiment can be used. Figure 3 The processing of the described generation unit 320 (in particular its determining sub-unit 3230) involves randomly determining at least a portion of the sample unit set from the 13 constructed sample unit sets Set1, Set2, Set3, ... Set13 to extract the minimum labeled sample.

[0201] In this case, for example, a reference can be used. Figure 3The processing of the described generation unit 320 involves repeatedly randomly determining at least a portion of the sample unit set, repeatedly extracting (e.g., randomly extracting) the minimum labeled sample unit from each determined sample unit set, and repeatedly generating labeled training samples based on the extracted minimum labeled sample units, thereby generating multiple labeled training samples to obtain a training sample set. This training sample set can be used, for example, directly as a second text sample set for a specific domain (or as part of a second text sample set) for training the recognition and classification model.

[0202] More preferably, in a second text sample set such as those obtained in the manner described above, the distribution of the number of indicators involved in each labeled training sample can conform to an expected distribution determined at least in part based on the original distribution of the number of indicators involved in multiple original samples in the original text sample set. Labeled training samples with such an expected distribution of the number of indicators involved can at least partially simulate the original distribution, enabling the classification and recognition model trained using it to achieve good performance when applied to real samples with the original distribution.

[0203] As an example, the aforementioned expected distribution can be determined at least in part based on the following normal distribution, the mean and variance of which are based on the mean and variance of the number of indicators involved in multiple original samples in the original text sample set. The specific form of this normal distribution can be, for example, determined by the original distribution of the number of indicators involved in multiple original samples in the original text sample set in accordance with the manner described above using formulas (1) to (4). Annotated training samples with such an expected distribution of the number of indicators involved effectively utilize the relatively balanced characteristics of the normal distribution to adjust for potential imbalances in the original samples, thereby enabling the classification and recognition models trained using these annotated training samples to be adequately and appropriately trained, and thus achieve good performance.

[0204] Furthermore, preferably, the aforementioned expected distribution can have the following form: in this expected distribution, the probability of occurrence of the number of indicators involved in each labeled training sample is equal to the weighted average of the first occurrence probability determined according to the normal distribution and the second occurrence probability determined according to the original distribution. Such a specific form of expected distribution is determined, for example, by adjusting the normal distribution using the original distribution of the number of indicators involved in multiple original samples in the original text sample set, as described above with reference to formulas (5) to (6). Labeled training samples with such an expected distribution of the number of indicators involved not only adjust for potential imbalances in the original samples but also avoid excessive differences from the original distribution. This allows the classification and recognition models trained using these labeled training samples to be adequately and appropriately trained with samples similar to the original distribution, thereby achieving good performance.

[0205] For example, the reference described above in the first embodiment can be used. Figures 4 to 5 The processing of the described generation unit 420 and its control subunit 4240 (and modules therein) ensures that the distribution of the number of metrics involved in each labeled training sample in the obtained training sample set conforms to the desired distribution, which will not be elaborated further here. In one example, a second text sample set, such as that obtained in the manner described above, may include a predetermined proportion of noise samples. For example, the above references can be used... Figure 6 The processing of the generation unit 620 described herein, which obtains a training sample set including a predetermined proportion of noise samples, serves as the second text sample set, will not be elaborated further here.

[0206] [4.3 Example Applications of Information Processing Equipment]

[0207] The information processing device according to the second embodiment, including a memory storing classification and recognition models, can be well applied in various vertical fields and can achieve good processing performance. The following will refer to... Figure 9 and Figure 10 An example application of the information processing device of the second embodiment is described.

[0208] Figure 9 An application scenario is shown where a classification and recognition model stored in the memory of an information processing device using the second embodiment is presented.

[0209] like Figure 9 As shown, a user review 910, used as input text, is fed into a classification and recognition model 920. The model 920 processes the text to produce an output result 930 (converted into a table format including text for ease of description). Here, user review 910 is a user's review of headphone products, for example, on a product page of an online store. The classification and recognition model 920 can identify portions of each of N=13 indicators (e.g., sound quality, noise cancellation, appearance, cost-effectiveness, Bluetooth performance, etc.) in the user review and classify the identified portions according to the relevant indicators, classifying them into one of M=3 sentiment categories (affirmative, negative, or neutral). The classification and recognition result 930 includes the identified indicator and the sentiment category determined for that indicator.

[0210] like Figure 9As shown, the classification and recognition model 920 includes a pre-trained word vector extraction model 921 and an untrained context representation model 922. The word vector extraction model 921 can be implemented based on the CWE model (i.e., based on the attention mechanism), and the context representation model 922 can be implemented using multiple Bi-LSTM layers, multiple LSTM layers, and a fully connected layer (i.e., based on a recurrent neural network). This structure is beneficial for improving the performance of the model.

[0211] Figure 10 The performance comparison of the classification and recognition models of Examples 1 and 2 according to this disclosure with the classification and recognition algorithm of Comparative Example 1 is shown.

[0212] The classification and recognition algorithm in Comparative Example 1 is implemented using a rule-based machine learning algorithm. In the classification and recognition model of Example 1 according to this disclosure, the pre-trained word vector extraction model is implemented using a word2vec model, while the untrained context representation model is implemented using a Bi-LTSM model. The classification and recognition model of Example 2 according to this disclosure uses... Figure 9 The structure of the classification and recognition model 920 is as follows: the pre-trained word vector extraction model is implemented using the BERT model (i.e., a CWE model based on an attention mechanism), while the untrained context representation model is implemented using multiple Bi-LSTM layers, multiple LSTM layers, and a fully connected layer. In other words, the word vector extraction model and the context representation model in Example 2 respectively employ an attention mechanism and an RNN model, thus the entire model adopts a "complementary" structure. Here, the various algorithms and models have comparable parameter sizes.

[0213] The task of the models or algorithms in the various examples and comparative examples is to identify N=13 indicators (e.g., sound quality, noise reduction, appearance, cost-effectiveness, Bluetooth performance, etc.) involved in user reviews, and to classify the identified portions of the reviews into one of M=3 sentiment categories (affirmative, negative, or neutral) based on the indicators involved. The models or algorithms in the various examples and comparative examples each used 3 million annotated user reviews as training data (this training data was generated in the manner described in [4.2 Sample Set for Training Models in Information Processing Devices], for example, using a domain-specific text sample set generated by an information processing device according to the first embodiment of this disclosure), and were tested using 2,000 real user reviews annotated with up to 13 indicators and corresponding categories.

[0214] like Figure 10As shown, Examples 1 and 2 of this disclosure outperform the algorithm of Comparative Example 1 in all aspects, including the number of False Negatives (FN), False Positives (FP), and True Positives (TP), as well as False Sentiment (FS), Precision, and Recall. The model of Example 2, which employs a structure combining an attention mechanism and a recurrent neural network, achieves significant improvements in all aspects. For example, the precision and recall of Example 1 increased from 71.6% and 70.7% in Comparative Example 1 to 77.8% and 80.6%, respectively, while the precision and recall of Example 2 further increased to 86.5% and 87.6%, respectively.

[0215] Note that in Figure 10 In the table shown, a false negative (FN) is defined as a situation where the relevant part of a user review is not identified, i.e., a missed detection. An example of this is as follows: a review sample "The sound quality is the best I've ever experienced in headphones," which is relevant to the sound quality metric and has a "positive" sentiment, is incorrectly identified as noise unrelated to the sound quality metric.

[0216] False positives (FP) are defined as user comments that do not involve relevant metrics but are incorrectly identified as relevant metrics and given a sentiment category. An example of this is a noise comment sample unrelated to sound quality, such as "Just received it, haven't tried the sound quality yet!", which is incorrectly identified as related to sound quality metrics and given a sentiment category of "positive".

[0217] True (TP) is defined as correctly identifying the metrics involved in a user review and correctly assigning a sentiment classification, i.e., the case of correct detection. An example of this is as follows: for a review sample "The high notes are very delicate and the low notes are also powerful" which is related to the sound quality metric and has a sentiment of "positive", it is correctly identified as related to the sound quality metric and assigned a sentiment classification of "positive".

[0218] False sentiment (FS) is defined as a situation where a user review correctly identifies the metrics involved but incorrectly assigns a sentiment category. An example of this is a review sample related to sound quality metrics, such as "The bass is decent, overall quality is acceptable," which is correctly identified as related to sound quality metrics but incorrectly assigned a "negative" sentiment category.

[0219] Furthermore, precision can be calculated based on the number of true positives (TP) and false positives (FP) as defined above, i.e., Precision = TP / (TP + FP). Additionally, recall can be calculated based on the number of true positives (TP) and false negatives (FN) as defined above, i.e., Recall = TP / (TP + FN).

[0220] The above describes a configuration example of the information processing apparatus according to a second embodiment of the present disclosure. As described above, by utilizing the information processing apparatus of the second embodiment, and in particular by utilizing the classification and recognition model stored therein, a word vector extraction model that is not specific to a particular domain and is pre-trained for text can be combined with an unpre-trained context representation model, thereby obtaining the former's good feature representation ability regarding word vectors and the latter's good context representation ability in a particular domain, thus improving the final recognition and classification results.

[0221] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that those skilled in the art will understand that all or any step or component of the methods and apparatus of this disclosure can be implemented in any computing device (including processors, storage media, etc.) or network of computing devices, in the form of hardware, firmware, software or a combination thereof. This is something that those skilled in the art can achieve by using their basic circuit design knowledge or basic programming skills after reading the description of this disclosure.

[0222] Furthermore, this disclosure also proposes a program product storing machine-readable instruction code. When the instruction code is read and executed by a machine, the methods described above according to embodiments of this disclosure can be performed.

[0223] Accordingly, the storage medium used to carry the program product storing machine-readable instruction code is also included in this disclosure. The storage medium includes, but is not limited to, floppy disks, optical disks, magneto-optical disks, memory cards, memory sticks, etc.

[0224] <5. Exemplary Structure of a General-Purpose Personal Computer for the Methods and / or Devices Used in the Embodiments>

[0225] In the case where this disclosure is implemented through software or firmware, transmission from a storage medium or network to a computer with a dedicated hardware architecture (e.g., Figure 11 The general-purpose computer 1100 shown is equipped with the programs that constitute the software, and when various programs are installed, the computer is able to perform various functions, etc.

[0226] Figure 11 This is a block diagram of an exemplary structure of a general-purpose personal computer in which methods and / or devices and / or systems according to embodiments of the present disclosure can be implemented.

[0227] exist Figure 11 In this configuration, the Central Processing Unit (CPU) 1101 performs various processes based on programs stored in the Read-Only Memory (ROM) 1102 or programs loaded into the Random Access Memory (RAM) 1103 from the Storage Section 1108. The RAM 1103 also stores data required as needed when the CPU 1101 performs various processes, etc. The CPU 1101, ROM 1102, and RAM 1103 are interconnected via a bus 1104. An Input / Output Interface 1105 is also connected to the bus 1104.

[0228] The following components are connected to input / output interface 1105: input section 1106 (including keyboard, mouse, etc.), output section 1107 (including display, such as cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.), storage section 1108 (including hard disk, etc.), and communication section 1109 (including network interface card, such as LAN card, modem, etc.). Communication section 1109 performs communication processing via a network, such as the Internet. Drive 1110 may also be connected to input / output interface 1105 as needed. Removable media 1111, such as disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1110 as needed, so that computer programs read from them are installed into storage section 1108 as needed.

[0229] When the above series of processes are implemented through software, the program constituting the software is installed from a network such as the Internet or a storage medium such as removable media 1111.

[0230] Those skilled in the art will understand that such storage media are not limited to Figure 11 The illustration shows a removable medium 1111 that stores a program and is distributed separately from the device to provide the program to the user. Examples of removable media 1111 include disks (including floppy disks (registered trademark)), optical disks (including optical disc read-only memory (CD-ROM) and digital versatile disks (DVD)), magneto-optical disks (including mini-discs (MD) (registered trademark)), and semiconductor memory. Alternatively, the storage medium may be ROM 1102, a hard disk included in storage section 1108, etc., containing programs and distributed to the user along with the device containing them.

[0231] Preferred embodiments of the present disclosure have been described above with reference to the accompanying drawings, but the present disclosure is by no means limited to the examples described above. Various changes and modifications can be made by those skilled in the art within the scope of the appended claims, and it should be understood that such changes and modifications naturally fall within the technical scope of the present disclosure.

[0232] For example, the units shown in the dashed boxes in the functional block diagrams shown in the attached figures represent that the functional unit is optional in the corresponding device, and the optional functional units can be combined in an appropriate manner to achieve the desired function.

[0233] For example, the multiple functions included in one unit in the above embodiments can be implemented by separate devices. Alternatively, the multiple functions implemented by multiple units in the above embodiments can be implemented by separate devices respectively. In addition, one of the above functions can be implemented by multiple units. Needless to say, such a configuration is included within the scope of the present disclosure.

[0234] In this specification, the steps described in the flowchart include not only processes executed sequentially in the stated order, but also processes executed in parallel or individually, rather than necessarily sequentially. Furthermore, even within the steps of sequential processing, needless to say, the order can be appropriately altered.

[0235] In addition, this disclosure may have the configuration described below.

[0236] (1) An information processing device, comprising:

[0237] The processing circuit is configured as follows:

[0238] For each of the multiple indicators, construct a sample unit set for that indicator based on multiple labeled minimum labeled sample units related to that indicator obtained from the original sample set; and

[0239] For at least a portion of the sample unit sets in the constructed multiple sample unit sets, extract the minimum labeled sample unit from each sample unit set, and generate labeled training samples based on the extracted minimum labeled unit.

[0240] (2) The information processing device as described in (1), wherein, for each of a plurality of indicators, the smallest labeling unit relating to the indicator is labeled with one of a plurality of predetermined labels for a classification task relating to the indicator.

[0241] (3) The information processing device as described in (1), wherein the processing circuit is further configured to:

[0242] Annotated training samples are generated by randomly combining the extracted smallest annotation units.

[0243] (4) The information processing device as described in (1), wherein the processing circuit is further configured to:

[0244] At least a portion of the sample unit sets are randomly selected from the constructed multiple sample unit sets.

[0245] (5) The information processing device as described in (4), wherein the processing circuit is further configured to:

[0246] Repeatedly perform the steps of randomly determining at least a portion of the sample unit set, extracting the smallest labeled sample unit, and generating labeled training samples to obtain a training sample set including multiple labeled training samples.

[0247] (6) The information processing device as described in (5), wherein the processing circuit is further configured to:

[0248] The desired distribution is determined at least in part based on the original distribution of the number of metrics involved in multiple original samples in the original sample set, and the repeated execution steps are controlled so that the distribution of the number of metrics involved in each labeled training sample in the obtained training sample set conforms to the desired distribution.

[0249] (7) The information processing device as described in (6), wherein,

[0250] The desired distribution is based, at least in part, on a normal distribution whose mean and variance are based on the mean and variance of the number of indicators involved in multiple original samples in the original sample set.

[0251] (8) The information processing device as described in (7), wherein the processing circuit is further configured to:

[0252] The desired distribution is determined by adjusting the normal distribution using the original distribution, such that the probability of occurrence of the number of indicators involved in each labeled training sample in the desired distribution is equal to the weighted average of a first occurrence probability determined according to the normal distribution and a second occurrence probability determined according to the original distribution.

[0253] (9) The information processing device as described in (5), wherein the processing circuit is further configured to:

[0254] Include a predetermined proportion of noise samples in the training sample set.

[0255] (10) The information processing apparatus as described in any one of (1) to (9), wherein the original sample set comprises a plurality of original text samples.

[0256] (11) The information processing device as described in (10), wherein each of the plurality of indicators is related to a specific content involved in the text sample, and each minimum annotation unit comprises a minimum set of consecutive sentences describing the corresponding indicator.

[0257] (12) An information processing device, comprising:

[0258] A memory storing domain-specific text recognition and classification models, said models being used to identify portions of domain-specific text input involving at least one metric, and to classify the identified portions with respect to the metric involved.

[0259] The identification and classification model includes:

[0260] A word vector extraction model, pre-trained using at least a first text sample set not specific to the stated domain, is used to extract word vectors from the text input; and

[0261] The contextual representation model, trained using a second text sample set of the specific domain, is used to output the classification result based on the word vectors extracted by the word vector extraction model.

[0262] (13) The information processing device as described in (12), wherein one of the word vector extraction model and the context representation model is based on a recurrent neural network and the other is based on an attention mechanism.

[0263] (14) The information processing apparatus as described in (12), wherein at least one labeled training sample in the second text sample set is obtained in the following manner:

[0264] For each of the multiple indicators, a sample unit set for that indicator is constructed based on multiple labeled minimum labeled sample units related to that indicator obtained from the original text sample set of the specific domain.

[0265] For at least a portion of the sample unit sets in the constructed multiple sample unit sets, extract the minimum labeled sample unit from each sample unit set, and generate labeled training samples based on the extracted minimum labeled unit.

[0266] (15) The information processing device as described in (14), wherein, for each of a plurality of indicators, the smallest labeling unit relating to the indicator is labeled with one of a plurality of predetermined labels for a classification task relating to the indicator.

[0267] (16) The information processing device as described in (14), wherein at least one labeled training sample is generated by randomly combining the extracted smallest labeled units.

[0268] (17) The information processing device as described in (14), wherein the at least part of the sample unit set is randomly determined from among the constructed multiple sample unit sets.

[0269] (18) The information processing device as described in (17), wherein, in the second text sample set, the distribution of the number of indicators involved in each labeled training sample conforms to an expected distribution determined at least in part based on the original distribution of the number of indicators involved in multiple original samples in the original text sample set.

[0270] (19) The information processing device as described in (18), wherein the expected distribution is determined at least in part according to a normal distribution whose mean and variance are based on the mean and variance of the number of indicators involved in a plurality of original samples in the original text sample set.

[0271] (20) The information processing device as described in (19), wherein, in the desired distribution, the probability of occurrence of the number of indicators involved in each labeled training sample is equal to the weighted average of a first probability of occurrence determined according to the normal distribution and a second probability of occurrence determined according to the original distribution.

[0272] (21) The information processing device as described in (14), wherein the second text sample set includes a predetermined proportion of noise samples.

[0273] (22) The information processing device as described in (14), wherein each of the plurality of indicators is related to specific content of the text in the particular domain, and each minimum annotation unit comprises a minimum set of consecutive sentences describing the corresponding indicator.

[0274] (23) An information processing method, comprising:

[0275] For each of the multiple indicators, construct a sample unit set for that indicator based on multiple labeled minimum labeled sample units related to that indicator obtained from the original sample set; and

[0276] For at least a portion of the sample unit sets in the constructed multiple sample unit sets, extract the minimum labeled sample unit from each sample unit set, and generate labeled training samples based on the extracted minimum labeled unit.

[0277] (24) A non-transitory computer-readable storage medium storing a program that, when executed by a processor, causes the processor to perform the method according to (23).

[0278] While embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, it should be understood that the embodiments described above are merely illustrative and do not constitute a limitation thereof. Those skilled in the art can make various modifications and alterations to the above embodiments without departing from the spirit and scope of the present disclosure. Therefore, the scope of the present disclosure is defined only by the appended claims and their equivalents.

Claims

1. An information processing device, comprising: The processing circuit is configured as follows: For each of the multiple indicators, a sample unit set for that indicator is constructed based on multiple labeled minimum labeled sample units related to that indicator obtained from the original sample set. The original sample set includes multiple original text samples, where each of the multiple indicators is related to the specific content involved in the text sample, and each minimum labeled sample unit includes the minimum set of continuous sentences describing the corresponding indicator. For at least a subset of sample unit sets from the constructed multiple sample unit sets, extract the smallest labeled sample unit from each sample unit set. In order to extract the smallest labeled sample unit, the processing circuit is also configured as follows: Extract the minimum set of continuous sentences containing feature keywords from the indicator feature library that describe the corresponding indicator; and Annotated training samples are generated from the extracted minimum labeled sample units in the form of the original sample set. In the preprocessing of extracting and labeling the minimum labeled sample units, the same labeling rules applicable to labeling the original sample set are used for the metrics. In order to generate labeled training samples, the processing circuit is also configured as follows: The extracted minimum labeled sample units are randomly combined, wherein the extracted minimum labeled sample units involve only the index of the given sample unit set and have only one label associated with that index.

2. The information processing device as described in claim 1, wherein, For each of the multiple metrics, the smallest labeled sample unit involving that metric is labeled with one of the multiple predefined labels for the classification task related to that metric.

3. The information processing device as described in claim 1, wherein, The processing circuit is also configured as follows: At least a portion of the sample unit sets are randomly selected from the constructed multiple sample unit sets.

4. The information processing device as described in claim 3, wherein, The processing circuit is also configured as follows: Repeatedly perform the steps of randomly determining at least a portion of the sample unit set, extracting the smallest labeled sample unit, and generating labeled training samples to obtain a training sample set including multiple labeled training samples.

5. The information processing device as described in claim 4, wherein, The processing circuit is also configured as follows: The desired distribution is determined at least in part based on the original distribution of the number of metrics involved in multiple original samples in the original sample set, and the repeated execution steps are controlled so that the distribution of the number of metrics involved in each labeled training sample in the obtained training sample set conforms to the desired distribution.

6. The information processing device as described in claim 5, wherein, The desired distribution is based, at least in part, on a normal distribution whose mean and variance are based on the mean and variance of the number of indicators involved in multiple original samples in the original sample set.

7. The information processing device as described in claim 6, wherein, The processing circuit is also configured as follows: The desired distribution is determined by adjusting the normal distribution using the original distribution, such that the probability of occurrence of the number of indicators involved in each labeled training sample in the desired distribution is equal to the weighted average of a first occurrence probability determined according to the normal distribution and a second occurrence probability determined according to the original distribution.

8. The information processing device as described in claim 4, wherein, The processing circuit is also configured as follows: Include a predetermined proportion of noise samples in the training sample set.

9. An information processing device, comprising: A memory storing domain-specific text recognition and classification models, said models being used to identify portions of domain-specific text input involving at least one metric, and to classify the identified portions with respect to the metric involved. The identification and classification model includes: A word vector extraction model, pre-trained using at least a first text sample set not specific to the stated domain, is used to extract word vectors from the text input; and A contextual representation model trained using a second text sample set from the specific domain is used to output the classification result based on the word vectors extracted by the word vector extraction model, wherein... At least one labeled training sample in the second text sample set was obtained in the following manner: For each of the multiple indicators, a sample unit set for that indicator is constructed based on multiple labeled minimum labeled sample units related to that indicator obtained from the original text sample set of the specific domain. Each of the multiple indicators is related to specific content covered by the text in the specific domain, and each minimum labeled sample unit includes a minimum set of consecutive sentences describing the corresponding indicator. For at least a subset of sample unit sets in the constructed multiple sample unit sets, extract the minimum labeled sample unit from each sample unit set, and generate labeled training samples based on the extracted minimum labeled sample unit. Extracting the smallest labeled sample unit from each sample unit set includes: Extract the smallest continuous set of sentences containing feature keywords from the indicator feature library that describe the corresponding indicators; The generated labeled training samples include: Annotated training samples are generated from the extracted minimum labeled sample units in the form of the original sample set. In the preprocessing for extracting and labeling the minimum labeled sample units, the same labeling rules applicable to the original sample set are used for the metrics. The extracted minimum labeled sample units are randomly combined, wherein the extracted minimum labeled sample units involve only the index of the given sample unit set and have only one label associated with that index.

10. The information processing device as described in claim 9, wherein, One of the word vector extraction model and the context representation model is based on a recurrent neural network, and the other is based on an attention mechanism.

11. The information processing device as claimed in claim 9, wherein, For each of the multiple metrics, the smallest labeled sample unit involving that metric is labeled with one of the multiple predefined labels for the classification task related to that metric.

12. The information processing device as described in claim 9, wherein, The at least part of the sample unit set is randomly determined from the multiple sample unit sets constructed.

13. The information processing device as claimed in claim 12, wherein, In the second text sample set, the distribution of the number of indicators involved in each labeled training sample conforms to the expected distribution determined at least in part based on the original distribution of the number of indicators involved in multiple original samples in the original text sample set.

14. The information processing apparatus as described in claim 13, wherein, The expected distribution is determined at least in part according to the following normal distribution, the mean and variance of which are based on the mean and variance of the number of indicators involved in multiple original samples in the original text sample set.

15. The information processing apparatus as described in claim 14, wherein, In the expected distribution, the probability of occurrence of the number of indicators involved in each labeled training sample is equal to the weighted average of the first occurrence probability determined according to the normal distribution and the second occurrence probability determined according to the original distribution.

16. The information processing device as claimed in claim 9, wherein, The second text sample set includes a predetermined proportion of noise samples.

17. An information processing method, comprising: For each of the multiple indicators, a sample unit set for that indicator is constructed based on multiple labeled minimum labeled sample units related to that indicator obtained from the original sample set. The original sample set includes multiple original text samples, where each of the multiple indicators is related to specific content in the text samples, and each minimum labeled sample unit includes the minimum set of consecutive sentences describing the corresponding indicator; and For at least a subset of the constructed sample unit sets, extract the minimum labeled sample unit from each sample unit set, and generate labeled training samples based on the extracted minimum labeled sample unit. Extracting the smallest labeled sample unit from each sample unit set includes: Extract the smallest continuous set of sentences containing feature keywords from the indicator feature library that describe the corresponding indicators; The generated labeled training samples include: Annotated training samples are generated from the extracted minimum labeled sample units in the form of the original sample set. In the preprocessing for extracting and labeling the minimum labeled sample units, the same labeling rules applicable to the original sample set are used for the metrics. The extracted minimum labeled sample units are randomly combined, wherein the extracted minimum labeled sample units involve only the index of the given sample unit set and have only one label associated with that index.

18. A non-transitory computer-readable storage medium storing a program, which, when executed by a processor, causes the processor to perform the method according to claim 17.

Images