Method, apparatus and computer-readable medium for identifying micro biased text within open corpora and generating responese to identified micro biased text

Abstract

The present invention is characterized by a method for identifying finely biased text within a public corpus, comprising: a natural language sentence collection step of collecting natural language sentences from a text-based dataset including a public conversation dataset and a web corpus dataset; a bias candidate dataset derivation step of deriving a bias candidate dataset including seed bias data expected to be socially and ethically biased and derived bias data extended from seed bias data by comparing semantic vectors between words constituting the natural language sentences using a word embedding model in the collected natural language sentences; and a bias determination step of determining the bias of the bias candidate dataset as one of a surface bias dataset, a finely biased dataset, and an unbiased dataset using a plurality of large-scale language models.

Description

Technology Field

[0001] The present invention relates to a method for identifying text with micro-biases within a public corpus and generating corresponding responses for the identified texts with micro-biases. Specifically, it relates to a technology that identifies texts with latent biases that are not outwardly apparent in a public corpus to construct data on micro-biases, and performs training on this data, thereby resolving the issue of reliability and performance degradation of a large-scale language model caused by the model performing incorrect bias detection. Background Technology

[0002] Along with the recent advancement of artificial intelligence technology, natural language processing technology is also developing rapidly. In particular, among the various models for neural machine translation, the performance of natural language tasks has dramatically improved with the release of translators applying self-attention and multi-head attention technologies. The BERT model, which uses only the encoder block of the translator, contributed significantly to the resurgence of deep learning technology for natural language processing, and GPT3, which uses only the decoder block, opened a new chapter in natural language generation by artificial intelligence through learning from massive corpora.

[0003] However, the advancement of artificial intelligence technology in the field of natural language processing (i.e., large-scale language models) has led to ethical issues regarding AI, such as the 'Iruda controversy.' Specifically, AI trained on various hate speech, personal information, and politically or ethically biased data mechanically provided biased predictions and results without any sense of guilt. This problem not only became a fatal weakness in the reliability of large-scale language models but also frequently posed a significant constraint on their commercialization.

[0004] Accordingly, Korean published patent No. 10-2023-0075890 (Device and method for outputting a language model with bias removed) proposes a language model output technology that can remove bias judgment and generated bias information by removing a module so that bias can be removed through human intervention in the deep learning process.

[0005] Meanwhile, the aforementioned prior art is a technology that eliminates bias by comparing and reviewing biased information generated by a large-scale language model with key information constructed through human intervention. However, it had limitations in that it could not resolve the problem of reduced reliability of the large-scale language model due to the high possibility of human error resulting from human intervention and the difficulty in identifying subtle biases that are not outwardly apparent, even though it is easy to identify biases that are outwardly apparent. The problem to be solved

[0006] Accordingly, the first objective of the present invention is to provide a technology that identifies text having superficially hidden micro-biases (or latent biases) corresponding to biases that large-scale language models cannot easily distinguish in public corpora.

[0007] Furthermore, the present invention has a second objective of providing a technology that enhances the fairness, ethics, and reliability of a large-scale language model by generating and providing an unbiased response when a question regarding finely biased text identified in the large-scale language model is received. means of solving the problem

[0008] To achieve the above-mentioned objective, a method for identifying finely biased text within a public corpus, implemented by a computing device comprising one or more processors and one or more memories for storing instructions executable by said processors according to one embodiment of the present invention, is characterized by comprising: a natural language sentence collection step of collecting natural language sentences from a text-based dataset including a public conversation dataset and a web corpus dataset; a bias candidate dataset derivation step of deriving a bias candidate dataset expected to be socially and ethically biased by comparing semantic vectors between words constituting the natural language sentences using a word embedding model in the collected natural language sentences; and a bias determination step of determining the bias of the bias candidate dataset as one of a superficial bias dataset, a finely biased dataset, and an unbiased dataset using a plurality of large-scale language models.

[0009] At this time, for the aforementioned bias judgment step, it is desirable to use heterogeneous large-scale language models having different structures and training mechanisms as multiple large-scale language models.

[0010] In addition, regarding the aforementioned bias judgment step, bias judgment is performed by multiple large-scale language models on a single bias candidate dataset that is the subject of bias judgment, and it is desirable to classify the single bias candidate dataset as a biased dataset if all large-scale language models determine that the single bias candidate dataset is a biased dataset.

[0011] In addition, regarding the aforementioned bias judgment step, bias judgment is performed by multiple large-scale language models on a single-bias candidate dataset that is the subject of bias judgment, and it is desirable to classify the single-bias candidate dataset as an unbiased dataset if all large-scale language models determine that the single-bias candidate dataset is an unbiased dataset.

[0012] In addition, the aforementioned bias judgment step performs bias judgment on a single-bias candidate dataset subject to bias judgment using multiple large-scale language models, and if at least one of the multiple large-scale language models determines the single-bias candidate dataset to be a biased dataset, it is desirable to classify the single-bias candidate dataset as a fine-biased dataset.

[0013] In addition, regarding the aforementioned bias judgment step, it is desirable to perform bias judgments on a single-bias candidate dataset subject to bias judgment using multiple large-scale language models, and if the multiple large-scale language models provide different bias judgment results, classify the single-bias candidate dataset as a finely biased dataset if the majority judgment is biased, and classify it as an unbiased dataset if the majority judgment is unbiased, in accordance with the principle of majority rule.

[0014] In addition, it is desirable to further include: a judgment criterion query step for querying a large-scale language model that derived the classification result of the fine-biased dataset when a single-biased candidate dataset is classified as a fine-biased dataset by performing the aforementioned bias judgment step; a response validity judgment step for judging the validity of a returned response when a response is returned by the large-scale language model by performing the judgment criterion query step; and a training dataset storage step for classifying the fine-biased dataset from which a valid response was derived in the response validity judgment step as a training dataset for self-learning and storing it in a training database.

[0015] In addition, regarding the aforementioned response validity judgment step, it is desirable to determine a response as valid if an element containing at least one of keywords and concepts involving social or ethical issues is derived from the response returned by the large-scale language model.

[0016] In addition, it is desirable to further include a self-learning step in which, after performing the aforementioned bias judgment step, self-learning is performed by providing training data related to fine bias to multiple large-scale language models using a training dataset stored in a training database.

[0017] Meanwhile, a method for identifying finely biased text within a public corpus and generating a corresponding response for the identified finely biased text, implemented by a computing device comprising one or more processors and one or more memories for storing instructions executable by said processors according to another embodiment of the present invention, is characterized by comprising: a natural language sentence collection step of collecting natural language sentences from a text-based dataset including a public conversation dataset and a web corpus dataset; a bias candidate dataset derivation step of deriving a bias candidate dataset expected to be socially and ethically biased by comparing semantic vectors between words constituting the natural language sentences using a word embedding model in the collected natural language sentences; a bias determination step of determining the bias of the bias candidate dataset as one of a surface bias dataset, a finely biased dataset, and an unbiased dataset using a plurality of large-scale language models; and a corresponding response generation step of generating a corresponding response for the natural language sentences identified as a surface bias dataset and a finely biased dataset, wherein the corresponding response is generated in which the bias is mitigated or removed.

[0018] At this time, the aforementioned response generation step preferably generates the response in which the bias has been identified as a response in a format that includes at least one of a summary format or a detailed description format.

[0019] In addition, the aforementioned bias judgment step preferably includes: a bias judgment result receiving step for receiving bias judgment results for a single bias candidate dataset from multiple large-scale language models; a bias score calculation step for aggregating the received bias judgment results and calculating a bias score for the single bias candidate dataset according to the principle of majority rule; and a bias strength definition step for defining the strength of bias for the single bias candidate dataset based on the calculated bias score.

[0020] In addition, the aforementioned multiple large-scale language models have their respective confidence levels defined by a pre-established confidence level management model, and in the bias score calculation step, it is desirable to calculate the bias score by assigning the highest weight to the bias judgment result provided by the large-scale language model with the highest defined confidence level.

[0021] In addition, regarding the aforementioned bias intensity definition step, it is desirable to define the first bias level if the bias score calculated for a single bias candidate dataset is less than the threshold bias score based on a pre-set threshold bias score, and to define the second bias level if the bias score calculated for the single bias candidate dataset is greater than or equal to the threshold bias score.

[0022] In addition, the aforementioned response generation step preferably generates a first response, which is a response composed of correction information that corrects the bias, when the bias strength defined for a bias candidate dataset is at the first bias level.

[0023] In addition, the aforementioned response generation step preferably generates a second response, which is a response consisting of warning information warning of the bias along with correction information that corrects the bias, when the defined bias strength for a single bias candidate dataset is at the second bias level.

[0024] Meanwhile, an identification device for finely biased text within a public corpus, implemented as a computing device comprising one or more processors and one or more memories for storing instructions executable by said processors, is characterized by comprising: a natural language sentence collection unit that collects natural language sentences from a text-based dataset including a public conversation dataset and a web corpus dataset; a bias candidate dataset derivation unit that derives a bias candidate dataset expected to be socially and ethically biased by comparing semantic vectors between words constituting said natural language sentences using a word embedding model in the collected natural language sentences; and a bias determination unit that determines the bias of the bias candidate dataset as one of a superficial bias dataset, a finely biased dataset, and an unbiased dataset using a plurality of large-scale language models.

[0025] In addition, the device for identifying finely biased text within a public corpus and generating a corresponding response for the identified finely biased text, implemented as a computing device comprising one or more processors and one or more memories for storing instructions executable by said processors, is characterized by comprising: a text data collection unit that collects natural language sentences from a text-based dataset including a public conversation dataset and a web corpus dataset; a bias candidate dataset derivation unit that derives a socially and ethically biased candidate dataset by comparing semantic vectors between words constituting the natural language sentences using a word embedding model in the collected natural language sentences; a bias determination unit that determines the bias of the bias candidate dataset derived by the bias candidate dataset derivation unit as one of a superficially biased dataset, a finely biased dataset, and an unbiased dataset using a plurality of large-scale language models; and a corresponding response generation unit that generates a corresponding response for text data identified as a superficially biased dataset and a finely biased dataset by the bias determination unit, wherein the corresponding response generates a response in which the bias is mitigated or removed.

[0026] On the other hand, regarding a computer-readable recording medium, the computer-readable recording medium stores instructions for a computing device to perform the following steps, wherein the steps include: a natural language sentence collection step of collecting natural language sentences from a text-based dataset including a public conversation dataset and a web corpus dataset; a bias candidate dataset derivation step of deriving a bias candidate dataset expected to be socially and ethically biased by comparing semantic vectors between words constituting the natural language sentences using a word embedding model in the collected natural language sentences; and a bias determination step of determining the bias of the bias candidate dataset as one of a superficial bias dataset, a micro-bias dataset, and an unbiased dataset using a plurality of large-scale language models.

[0027] In addition, in a computer-readable recording medium according to another embodiment of the present invention, the computer-readable recording medium stores instructions for a computing device to perform the following steps, wherein the steps include: a natural language sentence collection step of collecting natural language sentences from a text-based dataset including a public conversation dataset and a web corpus dataset; a bias candidate dataset derivation step of deriving a bias candidate dataset expected to be socially and ethically biased by comparing semantic vectors between words constituting the natural language sentences using a word embedding model in the collected natural language sentences; a bias determination step of determining the bias in the bias candidate dataset as one of a surface bias dataset, a micro bias dataset, and an unbiased dataset using a plurality of large-scale language models; and a corresponding answer generation step of generating a corresponding answer for a natural language sentence identified as a surface bias dataset and a micro bias dataset, wherein the corresponding answer is generated in which the bias is mitigated or removed. Effects of the invention

[0028] According to one embodiment of the present invention, the present invention provides a technique for identifying text having superficially obscured micro-biases (or latent biases) corresponding to biases that large-scale language models cannot easily distinguish in a public corpus, thereby resolving the problem of reduced user reliability caused by the large-scale language model's failure to detect superficially obscured biases.

[0029] In addition, according to one embodiment of the present invention, the present invention functions to generate a corresponding response in which the bias is mitigated or eliminated for natural language sentences in which bias or micro-bias is identified, thereby helping to enhance the user experience by providing fair services to all users, as well as contributing to reducing social prejudice and discrimination. Brief explanation of the drawing

[0031] FIGS. 1, 9, and 10 are flowcharts of a method for identifying finely biased text within a public corpus according to an embodiment of the present invention. FIG. 2 is an example of a collection source for natural language sentences according to an embodiment of the present invention. FIG. 3 is a conceptual example of the classification of surface bias, micro-bias, and unbiased sentences according to one embodiment of the present invention. FIG. 4 is a conceptual diagram of a plurality of LLM models for classifying natural language sentences according to an embodiment of the present invention. FIGS. 5 to 8 are examples in which bias judgment on natural language sentences is performed by a plurality of LLMs according to an embodiment of the present invention. FIGS. 11 and 13 are flowcharts of a method for identifying finely biased text within a public corpus and generating corresponding answers for the identified finely biased text according to an embodiment of the present invention. FIG. 12 is an example of a generation format of a corresponding answer according to an embodiment of the present invention. FIG. 14 is an example of a generation structure for a corresponding answer according to an embodiment of the present invention. FIG. 15 is a configuration diagram of a device for identifying finely biased text within a public corpus according to one embodiment of the present invention. FIG. 16 is a configuration diagram of a device for identifying finely biased text within a public corpus and generating corresponding answers for the identified finely biased text according to an embodiment of the present invention. FIG. 17 is an example of the internal configuration of a computing device according to an embodiment of the present invention. Specific details for implementing the invention

[0032] Hereinafter, various embodiments and / or aspects are disclosed with reference to the drawings. For illustrative purposes, numerous specific details are disclosed in the following description to aid in a general understanding of one or more aspects. However, it will also be recognized by those skilled in the art that these aspects may be practiced without such specific details. The following description and the accompanying drawings describe specific exemplary aspects of one or more aspects in detail. However, these aspects are exemplary, and some of the various methods in the principles of the various aspects may be used, and the description is intended to include all such aspects and their equivalents.

[0033] As used herein, terms such as "examples," "examples," "aspects," "examples," etc., may not be interpreted as implying that any aspect or design described is better or more advantageous than other aspects or designs.

[0034] Additionally, the terms “comprising” and / or “comprising” should be understood to mean that the relevant feature and / or component is present, but not to exclude the presence or addition of one or more other features, components and / or groups thereof.

[0035] Additionally, terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0036] Furthermore, in the embodiments of the present invention, all terms used herein, including technical or scientific terms, unless otherwise defined, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in the embodiments of the present invention.

[0037] The present invention relates to a method for identifying text with micro-biases within a public corpus and generating a corresponding response for the identified text with micro-biases. Specifically, the first objective is to provide a technology for identifying texts having superficial micro-biases (or latent biases) corresponding to biases that large-scale language models cannot easily distinguish in a public corpus. The second objective is to provide a technology for enhancing the fairness, ethics, and reliability of large-scale language models by generating and providing an unbiased corresponding response when a question regarding text with micro-biases identified by the large-scale language model is received.

[0038] Hereinafter, a detailed description of the present invention for achieving the above objectives will be provided with reference to the attached drawings, and multiple drawings may be simultaneously referenced to describe one or more technical features or components constituting the invention.

[0039] First, we will provide an explanation of the present invention with reference to Fig. 1, which illustrates a flowchart of a method for identifying finely biased text within a public corpus.

[0040] As illustrated in FIG. 1, a method for identifying finely biased text within a public corpus according to one embodiment of the present invention includes a natural language sentence collection step (S10) for collecting natural language sentences from a text-based dataset including a public conversation dataset and a web corpus dataset.

[0041] At this stage, in step S10, natural language sentences are collected from the publicly available Dehwa dataset and the web corpus dataset as described above.

[0042] Referring to FIG. 2 as an example embodiment, the conversation dataset described above can be understood as a text-based dataset consisting of conversations between people, such as chat records, customer service conversations, and forum threads, and the web corpus dataset described above is a various text-based dataset collected on the web, which may include news, web pages, blog posts, etc.

[0043] Meanwhile, these conversation datasets and web corpus datasets are stored in the first database (100) and the second database (110), respectively, and the respective databases are periodically updated so that natural language sentences reflecting recent trends can be collected, and the present invention is not limited thereto.

[0044] Returning to the description of Fig. 1, after the execution of the aforementioned step S10, a bias candidate dataset derivation step (S11) is performed, in which a bias candidate dataset including seed bias data expected to be socially and ethically biased and derived bias data extended from the aforementioned seed bias data is derived by comparing semantic vectors between words constituting the natural language sentences using a word embedding model in the collected natural language sentences.

[0045] As an example of one embodiment, the word embedding model mentioned in step S20 refers to a method of representing words as dense vectors in a high-dimensional vector space, and means a technique for converting words into a format that a computer can understand in natural language processing.

[0046] These word embedding models aim to represent semantic relationships between words in a vector space, where words with similar meanings are represented by similar vectors, and words with different meanings are represented by different vectors.

[0047] As an example of one embodiment, the word embedding model mentioned in the present invention may be trained through a neural network-based model. As a specific example, the algorithm of the word embedding model may use an algorithm that includes at least one of Word2Vec, GloVe (Global vectors for word representation), and FastText.

[0048] At this time, the aforementioned Word2Vec is a neural network-based word embedding algorithm that uses a model structure including at least one of a model structure that predicts a central word based on the context of surrounding words (CBOW, Continuous Bag of words) and a model structure that predicts surrounding words based on the central word (Skip-gram), and can express semantic similarity between words as a vector by learning the relationships between words surrounding a specific word.

[0049] Furthermore, the aforementioned GloVe learns relationships between words in a global context using a word co-occurrence matrix. As an example, while the aforementioned Word2Vec focuses on the central word, GloVe collects the frequency of words appearing together in entire sentences to reflect overall statistical relationships, and is characterized by a structure in which word vectors are learned through this process.

[0050] In addition, the aforementioned FastText is characterized by effectively performing morphological analysis by learning words by dividing them into character units (tokens). Accordingly, it can flexibly handle the calculation of similarity for neologisms or compound words that have not been previously learned.

[0051] Meanwhile, in the aforementioned S11 step, a process of converting each word constituting a natural language sentence into a vector using a word embedding model is performed first. At this time, the converted vector reflects the meaning the word has in the context and can be usefully utilized to analyze semantic similarity and relationships between words.

[0052] As an example, vectors generated through word embeddings can be mathematically compared. For instance, the similarity between vectors can be measured using methods such as cosine similarity or Euclidean distance, and the relationship between words can be analyzed through this. In the case of cosine similarity, the angle between words is compared, and the closer it is to 0, the more similar the two words are; and in the case of Euclidean distance, the straight-line distance between vectors is calculated, and the closer the distance, the more similar the two words are.

[0053] In this invention, abnormally strong semantic associations may be detected between words related to specific social groups, gender, race, or religion; for example, if "female" and "emotional" are represented by very close vectors in a natural language sentence, the natural language sentence may be judged to have a bias reflecting gender stereotypes.

[0054] That is, in step S11 of the present invention, it can be understood that a process is performed to determine whether there are keywords in natural language sentences that are related to specific social groups, gender, race, or religion and are likely to cause social / ethical problems, and to derive them as a bias candidate dataset.

[0055] The aforementioned keywords that may cause social / ethical problems may be identified by comparing them with predefined keywords, where keywords related to social groups, gender, race, and religion are defined in advance, and the present invention is not limited thereto.

[0056] In particular, the aforementioned step S11 is characterized by collecting sentences detected as having bias from the natural language sentences collected in step S10 as seed bias data, and then expanding the collected seed bias data to derive derived bias data. At this time, the aforementioned derived bias data can be understood as a value derived from sentences with similar meanings that were not directly collected as seed bias data, but were derived by applying the seed bias data to a word embedding model to expand its meaning.

[0057] Specifically, the derived bias data derived by expanding the seed bias data can be understood as replacing a specific word in the seed bias data with another word having a similar meaning, reconstructing a specific sentence in the seed bias data to have a similar meaning, or adding additional data to the seed bias data to reflect various perspectives and opinions. In the present invention, by deriving bias candidate data using the seed bias data and the derived bias data as described above, the accuracy of bias judgment can be improved and the effect of self-feedback can be expected.

[0058] Next, after the execution of the aforementioned step S11, a bias determination step (S12) is performed in which the bias of the bias candidate dataset is determined to be one of a surface bias dataset, a fine bias dataset, or an unbiased dataset using a plurality of large-scale language models.

[0059] Generally, to detect whether bias exists in a natural language sentence, one examines whether a topic containing discriminative elements is explicitly or implicitly included within the sentence; if explicit elements are included, it is judged to be a natural language sentence with surface bias, if implicit elements are included, it is judged to be a natural language sentence with fine bias, and if neither explicit nor implicit elements are included, it is judged to be an unbiased natural language sentence.

[0060] Referring to FIG. 3 as an example, FIG. 3 (A), (B), and (C) illustrate examples of natural language sentences with surface bias, natural language sentences with fine bias, and unbiased natural language sentences.

[0061] First, referring to (A) in Figure 3, regarding the natural language sentence “Asians are good at math but lack social skills,” it can be seen that it explicitly contains many keywords containing stereotypes related to race, and thus it can be classified as a natural language sentence containing superficial bias.

[0062] Next, referring to (B) in Fig. 3, the natural language sentence “He is white but he raps really well” appears, and since this natural language sentence implies that it is unusual for a specific race to rap well, it can be classified as a natural language sentence with fine bias in that it contains an ability stereotype about a specific race.

[0063] Next, referring to (C) in Fig. 3, a natural language sentence appears stating, "People can perform various roles depending on their personality and abilities." Since this natural language sentence does not contain stereotypes regarding a specific gender, race, religion, or social group, it can be classified as an unbiased natural language sentence.

[0064] Meanwhile, in step S12 of the present invention, bias judgment is performed on a bias candidate dataset that is subject to bias judgment using a plurality of large-scale language models based on these bias judgment criteria, and more preferably, heterogeneous large-scale language models having different structures and training mechanisms are used as shown in the conceptual diagram in Fig. 4.

[0065] As an example of one embodiment, when two types of large-scale language models are used, bias in natural language sentences can be determined by using heterogeneous large-scale language models, such as a BERT-based large-scale language model and a GPT-based large-scale language model, each having different learning methods and text processing methods.

[0066] Large-scale language models based on BERT utilize bidirectional learning centered on specific words to consider the context surrounding a sentence, thereby deeply understanding the meaning of words depending on the context. In contrast, large-scale language models based on GPT utilize unidirectional learning to sequentially generate words and predict sentences from the beginning. Consequently, they can efficiently perform biased analysis related to the natural flow of sentences.

[0067] As an example, a BERT-based large-scale language model demonstrates excellent performance in identifying contextual biases (i.e., detecting implicit biases) that reflect subtle stereotypes about women by linking "delicate" and "emotional" in the natural language sentence "Women are delicate and therefore emotional," while a GPT-based large-scale language model demonstrates excellent performance in identifying biases arising during the sentence generation process (easily detecting implicit or pattern biases) such as "it is rare for women" by learning the natural flow of words in the natural language sentence "She succeeded as a CEO, but this is rare for women."

[0068] In other words, the present invention utilizes the characteristics of the aforementioned BERT-based large-scale language model, which deeply understands context and effectively detects subtle contextual biases, and the GPT-based large-scale language model, which more efficiently identifies biases occurring in the sentence generation flow, in a complementary manner. This enables the identification of various aspects of bias that are difficult to detect with a single large-scale language model with high accuracy, thereby allowing for the construction of a better ethical artificial intelligence system and the provision of services that fulfill social responsibility.

[0069] Meanwhile, in step S12 of the present invention, a bias judgment is performed on a bias candidate dataset subject to bias judgment using a plurality of large-scale language models, and detailed classification criteria may be provided for classifying whether the bias candidate dataset is superficial bias, micro-bias, or unbiased.

[0070] In one embodiment, the present invention performs a bias judgment on a bias candidate dataset subject to bias judgment using a plurality of large-scale language models, wherein if all large-scale language models determine the bias candidate dataset as a biased dataset, the bias candidate dataset can be classified as a surface bias dataset.

[0071] To explain the concept with reference to the figure in Fig. 5, for example, regarding a natural language sentence (T1) that is subject to bias judgment, when performing bias judgment, a plurality of large-scale language models, such as 'a', 'b', and 'c', are used, and when these large-scale language models determine that there is bias in the natural language sentence (T1), step S12 of the present invention can classify the natural language sentence (T1) as a dataset with surface bias.

[0072] In addition, as an example embodiment, the present invention performs a bias determination on a bias candidate dataset subject to bias determination using a plurality of large-scale language models, wherein if all large-scale language models determine the bias candidate dataset as an unbiased dataset, the bias candidate dataset can be classified as a dataset that is not biased, i.e., an unbiased dataset.

[0073] To explain the concept with reference to the figure in Fig. 8, for example, regarding a natural language sentence (T1) subject to bias judgment, when performing bias judgment, a plurality of large-scale language models, such as 'a', 'b', and 'c', are used, and if these large-scale language models determine that the natural language sentence (T1) is unbiased, then step S12 of the present invention can classify the natural language sentence (T1) as an unbiased dataset.

[0074] In addition, as an embodiment, the present invention performs a bias judgment on a bias candidate dataset subject to bias judgment in a plurality of large-scale language models, and if at least one of the plurality of large-scale language models determines the bias candidate dataset as a biased dataset, the bias candidate dataset can be classified as a finely biased dataset.

[0075] To explain the concept with reference to the figure in Fig. 6, for example, when performing a bias judgment on a natural language sentence (T1) that is subject to bias judgment, a plurality of large-scale language models, such as 'a', 'b', and 'c', are used. Among these large-scale language models, if 'a' determines that the natural language sentence (T1) is a biased dataset, and 'b' and 'c' determine that the natural language sentence (T1) is a non-biased dataset, then the present invention classifies the natural language sentence (T1) as a finely biased dataset.

[0076] That is, in the present invention, when at least one of a plurality of large-scale language models determines that a natural language sentence (T1) subject to bias judgment has a bias, the natural language sentence (T1) is determined to have a potential for bias, thereby making it easy to identify a dataset with subtle bias that is not outwardly apparent.

[0077] Of course, in another embodiment of the present invention, when identifying fine bias, bias judgment is performed by each of the multiple large-scale language models on a natural language sentence (T1), which is a bias candidate dataset subject to bias judgment. When the multiple large-scale language models provide different bias judgment results, the bias candidate dataset may be classified as a fine bias dataset if the bias judgment is majority according to the principle of majority rule, and the bias candidate dataset may be classified as a non-bias dataset if the non-bias judgment is majority.

[0078] As an example of one embodiment, referring to FIG. 7, FIG. 7 illustrates a conceptual diagram of an example in which a natural language sentence (T1) is classified as a dataset with fine bias as a result of performing bias judgment on a natural language sentence (T1) that is subject to bias judgment in a plurality of large-scale language models, and the plurality of large-scale language models make a bias judgment on the natural language sentence (T1).

[0079] In this case, in another embodiment of the present invention, reliability management for a plurality of large-scale language models may be performed so that a bias judgment reflecting the majority rule principle and the reliability of each large-scale language model is performed.

[0080] In other words, it is possible to derive bias judgment results for natural language sentences from multiple large-scale language models by assigning different weights according to the reliability evaluated by the models, such as assigning the highest weight to the judgment result provided by the large-scale language model with the highest reliability and the lowest weight to the judgment result provided by the large-scale language model with the lowest reliability.

[0081] Meanwhile, the reliability of the aforementioned large-scale language model can be evaluated and managed by the following evaluation means.

[0082] As an example of one embodiment, the present invention may perform an accuracy evaluation of a large-scale language model to determine bias based on the reliability of the large-scale language model. For example, to determine whether the response of the large-scale language model is factually accurate, a problem with an answer key may be provided to the large-scale language model, and then the reliability may be evaluated by determining how closely the response returned by the large-scale language model matches the actual answer key.

[0083] In addition, as an example embodiment, the present invention may perform a consistency evaluation of a large-scale language model to determine bias based on the reliability of the large-scale language model. For instance, this involves evaluating whether the large-scale language model provides consistent answers to the same question or in similar contexts, and the reliability can be evaluated by determining whether the large-scale language model provides consistent answers when repeated questions are asked in the same situation or when the way the question is expressed is changed slightly.

[0084] In addition, as an exemplary embodiment, the present invention may perform a stability evaluation to determine bias based on the reliability of a large-scale language model. For instance, this is intended to examine how well the large-scale language model responds to variations in input; for instance, the reliability can be evaluated by verifying whether the large-scale language model still returns a correct response or derives and returns a meaningful response even in the event of inputs containing typos or grammatical errors.

[0085] In addition, as an exemplary embodiment, the present invention may evaluate whether a response is factual in order to perform bias judgment based on the reliability of a large-scale language model. This evaluation assesses how closely the response returned by the large-scale language model matches the facts, and the reliability can be evaluated by verifying whether the response returned by the large-scale language model is actually based on a reliable source.

[0086] In addition, as an example embodiment, the present invention may perform a user satisfaction evaluation to determine bias based on the reliability of a large-scale language model. This evaluation method assesses how satisfied actual users of the large-scale language model are with the performance of the large-scale language model, and can evaluate the reliability by assessing how practically helpful the response of the large-scale language model was in solving the problem.

[0087] Of course, in the present invention, the reliability of a large-scale language model may be evaluated and managed using one of the evaluation methods above; however, preferably, the reliability evaluation may be performed and managed using two or more evaluation methods or by combining all evaluation methods to determine which large-scale language model to assign a relatively high weight and which to assign a relatively low weight, and the present invention is not limited thereto.

[0088] Meanwhile, if a one-biased candidate dataset is classified as a fine-biased dataset in step S12 of Fig. 1, an additional process may be performed to verify the classified fine-biased dataset.

[0089] Referring to FIG. 9 for a specific explanation of this, the present invention performs a judgment criterion query step (S121) in which a judgment criterion is queried to a large-scale language model that derives a classification result of a fine bias dataset, and then, if there is a response returned by the large-scale language model in step S121, a response validity judgment step (S122) in which the validity of the returned response is judged can be performed.

[0090] At this time, in the aforementioned S122 step, a response is determined to be valid if an element containing at least one of keywords and concepts including social or ethical issues is derived from the response returned by the large-scale language model, and, for example, it may be a concept containing keywords and concepts in areas such as gender, race, religion, sexual orientation, and social status.

[0091] As a specific example, regarding keywords related to race, one can cite cases where keywords related to "crime," "violent," or "laziness" are included for "Black people," or where keywords related to "study," "mathematics," or "obedient" are included for "Asian people."

[0092] That is, in the present invention, at step S122, a large-scale language model determines that a candidate dataset for one bias has a fine bias, and if a valid reason is derived among the reasons for the large-scale language model's determination of fine bias, such as due to keywords and concepts including social and ethical issues, a final verification is performed to confirm that the dataset primarily determined to have a fine bias is validly classified as a fine-biased dataset.

[0093] In addition, as illustrated in Fig. 9, the fine bias dataset from which a valid response is derived in step 122 is further classified as a learning dataset for self-learning and stored in a learning database, and a learning dataset storage step (S123) is included.

[0094] In other words, the present invention allows for the construction of a dataset for identifying fine bias data by utilizing a fine bias dataset from which valid reasons have been derived as training data.

[0095] Accordingly, after performing step S12 including steps S121 to S123, a self-learning step (S13) is performed to enable self-learning by providing training data related to fine bias to multiple large-scale language models using a training dataset stored in a training database.

[0096] In other words, step S13 can be understood as a concept of post-training a pre-trained large-scale language model using a fine bias dataset stored in a training database, and by adding new knowledge related to fine bias to the already learned knowledge related to fine bias, it can contribute to enabling the large-scale language model to generate more sophisticated responses.

[0097] In particular, in the present invention, by using a fine bias dataset from which valid reasons have been derived as a training dataset, the occurrence of incorrect learning regarding fine biases by a large-scale language model during self-learning can be minimized, thereby significantly increasing the efficiency of self-learning and, accordingly, achieving performance improvement with a large-scale language model that has enhanced fine bias identification performance.

[0098] Meanwhile, the present invention goes further than providing a method for identifying finely biased text within the aforementioned public corpus to propose a method for generating a corresponding response for the identified finely biased text.

[0099] Specifically, referring to FIG. 11, the description of the present invention can be performed in a natural language sentence collection step (S20) that collects natural language sentences from a text-based dataset including a publicly available conversational data set and a web corpus dataset.

[0100] In this case, the aforementioned S20 step may be understood as being able to provide all the functions and effects performed by the S10 step mentioned in the method for identifying finely biased text within the aforementioned public corpus.

[0101] Additionally, after performing the above-described step S20, the method includes a bias candidate dataset derivation step (S21) for deriving a bias candidate dataset expected to be socially and ethically biased by comparing semantic vectors between words constituting the natural language sentences using a word embedding model in the collected natural language sentences.

[0102] In this case, the aforementioned S21 step may be understood as being able to provide all the functions and effects performed by the S11 step mentioned in the method for identifying finely biased text within the aforementioned public corpus.

[0103] In addition, after the execution of the aforementioned step S21, a bias determination step (S22) is performed in which the bias of the bias candidate dataset is determined to be one of a surface bias dataset, a micro-bias dataset, and an unbiased dataset using a plurality of large-scale language models.

[0104] As an example, the above-described step S22 can be understood to provide all the functions and effects performed by step S12 mentioned in the method for identifying finely biased text within the above-described public corpus.

[0105] In addition, as another embodiment, the above-described step S22 may undergo the following processing steps to determine bias for a biased candidate dataset.

[0106] With reference to FIG. 13, the explanation continues in step S22, which involves receiving bias judgment results for a bias candidate dataset from multiple large-scale language models (S221), then performing a bias score calculation step (S222) that aggregates the received bias judgment results and calculates a bias score for the bias candidate dataset according to the principle of majority rule, and then performing a bias strength definition step (S223) that defines the bias strength for the bias candidate dataset based on the calculated bias score.

[0107] At this time, the bias score mentioned in step S222 can be processed by assigning 10 points if it is determined to have surface bias, 5 points if it is determined to have micro-bias, and 0 points if it is determined to be an unbiased dataset, and then dividing the bias score aggregated in the bias candidate dataset by the number of large-scale language models participating in the bias judgment to calculate the average bias score.

[0108] For example, in the present invention, if the average score of bias is 0 points, the one-bias candidate dataset is determined to be an unbiased natural language sentence; if the average score of bias is 10 points, the one-bias candidate dataset is determined to be a natural language sentence with apparent bias; and if the average score of bias is between 1 and 9 points, the one-bias candidate dataset is determined to be a natural language sentence with fine bias.

[0109] Of course, in another embodiment of the present invention, when performing step S222, the bias score may be calculated by assigning the highest weight to the bias judgment result provided by the large-scale language model with the highest defined confidence level according to the confidence level management model pre-set for a plurality of large-scale language models, and in the case of the large-scale language model with the lowest defined confidence level, the bias score may be calculated by assigning the lowest weight corresponding to the highest weight, and the present invention is not limited thereto.

[0110] Of course, it is desirable to understand that these embodiments are examples in which different weights are applied according to the confidence level defined for each large-scale language model, and the means for evaluating the confidence level of these large-scale language models can be understood as having the same or similar mechanism as the aforementioned confidence level evaluation method.

[0111] Meanwhile, in the present invention, when a bias score is calculated in step S222 as described above, a bias strength definition step (S223) is performed to define the bias strength for a bias candidate dataset based on the calculated bias score.

[0112] At this time, the bias strength defined by step S223 is characterized by defining the first bias level when the bias score calculated for a single bias candidate dataset is less than the threshold bias score based on a pre-set threshold bias score, and defining the second bias level when the bias score calculated for a single bias candidate dataset is greater than or equal to the threshold bias score.

[0113] For example, the first bias level may mean the range of 1 to 5 points when the total bias score is assigned from 0 to 10 points, and the second bias level may mean 6 to 9 points or 6 to 10 points including surface bias, that is, the second bias level means a range where the intensity of the bias is higher than that of the first bias level.

[0114] Meanwhile, after the execution of step S22, if a bias candidate dataset is identified as a natural language sentence in either a surface bias dataset or a micro-bias dataset based on the bias judgment result in step S22, a corresponding answer generation step (S23) is performed to generate a corresponding answer for the identified natural language sentence, wherein the bias is mitigated or removed.

[0115] At this time, in the aforementioned step S23, a corresponding response to a natural language sentence in which a bias has been identified can be generated as a corresponding response in a format including at least one of a summary format or a detailed description format, and an example thereof will be explained with reference to FIG. 12.

[0116] Specifically, when a natural language sentence with identified bias is T1 in FIG. 12, a corresponding response in summary form can be generated and provided as a corresponding response that includes correction information for correcting the natural language sentence while providing a response to the biased natural language sentence as in A1, and a corresponding response in detailed description form can be generated and provided as a corresponding response that includes warning information warning that the natural language sentence is biased and correction information while providing a response to the natural language sentence as in A2.

[0117] In other words, summary-style responses are characterized by correcting biases in natural language sentences while designing the generation of corresponding responses so that the user does not perceive the content, whereas detailed-style responses are characterized by correcting biases in natural language sentences as well as designing the generation of corresponding responses that can provide an educational effect by enabling the user to recognize problems and biased expressions in the text they entered.

[0118] Meanwhile, in step S23, the method of providing a corresponding response can be varied depending on whether the defined bias strength for a single-bias candidate dataset is at the first bias level or the second bias level.

[0119] In one embodiment, the present invention may generate a first corresponding answer composed of correction information that corrects the bias when the bias strength defined for a bias candidate dataset is at a first bias level, and may generate a second corresponding answer composed of warning information that warns of the bias along with correction information that corrects the bias when the strength defined for a bias candidate dataset is at a second bias level.

[0120] Referring to Fig. 14 as a more specific example, when the bias intensity defined in the natural language sentence T1 in which the bias is identified is the first bias level, the first corresponding response can be generated and provided as a corresponding response A1 that includes only correction information for correcting the bias, and when the bias intensity defined in the natural language sentence T1 in which the bias is identified is the second bias level, the second corresponding response can be generated and provided as a second corresponding response A2 that consists of correction information for correcting the bias and warning information for warning the bias.

[0121] In other words, the present invention enables a large-scale language model to identify biased sentences and correct and warn against them, thereby fulfilling social responsibility and enabling the development of ethical artificial intelligence technology in a trustworthy direction, and providing fair services to users of the large-scale language model.

[0122] On the other hand, regarding the description of the method for identifying finely biased text within a public corpus and generating corresponding responses for the identified finely biased text, one or more bias candidate datasets classified as finely biased datasets by performing the above-described step S22 may be used to construct training data for self-training a large-scale language model by performing steps S121 to S123 mentioned in the method for identifying finely biased text within a public corpus, and then the step S13 of Fig. 10 may be additionally performed before or after the step S23 of Fig. 11 to enhance the finely biased identification ability of the large-scale language model, and the present invention is not limited thereto.

[0123] Next, we will present an explanation of the identification mechanism for finely biased text within the public corpus.

[0124] Referring to 10A of FIG. 15, the main configuration of a device for identifying finely biased text within a public corpus according to one embodiment of the present invention may include, as a main configuration, a natural language sentence collection unit (11), a bias candidate dataset derivation unit (12), and a bias determination unit (13).

[0125] Specifically, the natural language sentence collection unit (11) described above functions to collect natural language sentences from a first database (100) in which a public conversation dataset is stored and a second database (110) in which a web corpus dataset is stored. That is, the natural language sentence collection unit (11) described above can be understood as capable of performing all the functions performed by step S10 of FIG. 1 described above, and natural language sentences collected from various media can be collected through the performance of the functions of the natural language sentence collection unit (11).

[0126] In addition, the above-described bias candidate dataset derivation unit (12) functions to derive a bias candidate dataset including seed bias data expected to be socially and ethically biased and derived bias data expanded from seed bias data by comparing semantic vectors between words constituting natural language sentences using a word embedding model in collected natural language sentences.

[0127] That is, the above-described bias candidate dataset derivation unit (12) can be understood as capable of performing all the functions performed by the above-described step S11 of FIG. 1, and in the present invention, by performing the functions of such bias candidate dataset derivation unit (12), it is possible to determine whether there are keywords in natural language sentences that are likely to cause social / ethical problems as keywords related to specific social groups, gender, race, or religion.

[0128] In addition, the bias determination unit (13) described above performs a function of determining the bias of the bias candidate dataset derived from the bias candidate dataset derivation unit (12) using a plurality of large-scale language models as one of the biased dataset, the finely biased dataset, and the unbiased dataset.

[0129] That is, the bias judgment unit (13) described above can be understood as being capable of performing all the functions of the S12 step of FIG. 1 described above. By performing the functions of this bias judgment unit (13), not only can biases that are outwardly apparent in natural language sentences be easily identified, but subtle biases that are not outwardly apparent can also be easily identified, thereby enhancing the bias identification ability of the large-scale language model and increasing the reliability of the large-scale language model.

[0130] Meanwhile, although not explicitly shown in FIG. 15, the bias judgment unit (13) may further include a judgment criterion query unit (not shown) that queries a judgment criterion to a large-scale language model that derives a classification result of a fine-biased dataset when a single biased candidate dataset is classified as a fine-biased dataset, a response validity judgment unit (not shown) that determines the validity of a returned response when there is a response returned by the large-scale language model according to the function of the judgment criterion query unit, a learning dataset storage unit (not shown) that classifies the fine-biased dataset from which a valid response was derived as a learning dataset for self-learning and stores it in a learning database, and a self-learning unit (not shown) that enables the large-scale language model to self-learn using the learning data stored in the learning dataset storage unit.

[0131] At this time, the aforementioned judgment criteria query unit, response validity judgment unit, and learning dataset storage unit can be understood as being capable of performing all the functions mentioned in steps S121, S122, and S123 of FIG. 9, and the self-learning unit can be understood as being capable of performing all the functions mentioned in the description of step S13 of FIG. 10.

[0132] On the other hand, the present invention includes a device for identifying finely biased text within a public corpus and generating a corresponding response for the identified finely biased text.

[0133] Referring to 10B of FIG. 16, the main configuration of the device for identifying micro-biased text within a public corpus and generating corresponding answers for the identified micro-biased text is described. The device of the present invention may include a natural language sentence collection unit (11), a bias candidate dataset derivation unit (12), a bias determination unit (13), and a corresponding answer generation unit (14).

[0134] At this time, the natural language sentence collection unit (11) described above functions to collect natural language sentences from a first database (100) in which a public conversation dataset is stored and a second database (110) in which a web corpus dataset is stored. That is, the natural language sentence collection unit (11) described above can be understood as capable of performing all the functions performed by step S20 of FIG. 11 described above, and can collect natural language sentences collected from various media through the performance of the functions of the natural language sentence collection unit (11).

[0135] In addition, the above-described bias candidate dataset derivation unit (12) functions to derive a socially and ethically biased candidate dataset by comparing semantic vectors between words constituting natural language sentences using a word embedding model in collected natural language sentences.

[0136] That is, the above-described bias candidate dataset derivation unit (12) can be understood as being capable of performing all the functions performed by the above-described step S21 of FIG. 11, and in the present invention, by performing the functions of such bias candidate dataset derivation unit (12), it is possible to determine whether there are keywords in natural language sentences that are likely to cause social / ethical problems as keywords related to specific social groups, gender, race, or religion.

[0137] In addition, the bias determination unit (13) described above performs a function of determining the bias of the bias candidate dataset derived from the bias candidate dataset derivation unit (12) using a plurality of large-scale language models as one of the biased dataset, the finely biased dataset, and the unbiased dataset.

[0138] That is, the bias judgment unit (13) described above can be understood as being capable of performing all the functions of step S22 of FIG. 11 described above. By performing the functions of this bias judgment unit (13), not only can biases that are outwardly apparent in natural language sentences be easily identified, but subtle biases that are not outwardly apparent can also be easily identified, thereby enhancing the bias identification ability of the large-scale language model and increasing the reliability of the large-scale language model.

[0139] In addition, the above-described corresponding answer generation unit (14) generates a corresponding answer for the text data identified by the bias determination unit (13) as the surface bias dataset and the fine bias dataset, and functions to generate a corresponding answer in which the bias is mitigated or removed.

[0140] That is, the above-described corresponding answer generation unit (14) can be understood as being capable of performing all the functions performed by step S23 of FIG. 11 described above. In the present invention, by performing the functions of such a corresponding answer generation unit (14), it can not only help improve the user experience by providing fair service to all users, but also contribute to reducing social prejudice and discrimination.

[0141] Meanwhile, although not explicitly illustrated in FIG. 16, the device of the present invention may further include, as detailed configurations of the bias judgment unit (12), a bias judgment result receiving unit that receives bias judgment results for the one bias candidate dataset from a plurality of large-scale language models, a bias score calculation unit that aggregates the received bias judgment results and calculates a bias score for the one bias candidate dataset according to the principle of majority rule, and a bias strength definition unit that defines the strength of the bias for the one bias candidate dataset based on the calculated bias score, and this corresponds to steps S221, S222, and S223 of FIG. 13, and it can be understood that all functions mentioned in each step can be performed, but the present invention is not limited thereto.

[0142] According to one embodiment of the present invention described above, the present invention provides a technology for identifying text having superficially obscured micro-biases (or latent biases) corresponding to biases that large-scale language models cannot easily distinguish in a public corpus, thereby resolving the problem of reduced user reliability caused by the large-scale language model's failure to detect superficially obscured biases.

[0143] In addition, according to one embodiment of the present invention, the present invention functions to generate a corresponding response in which the bias is mitigated or eliminated for natural language sentences in which bias or micro-bias is identified, thereby helping to enhance the user experience by providing fair services to all users, as well as contributing to reducing social prejudice and discrimination.

[0144] Although the embodiments have been described above with reference to limited embodiments and drawings, those skilled in the art can make various modifications and variations from the description above.

[0145] On the other hand, referring to FIG. 17, FIG. 17 illustrates an example of the internal configuration of a computing device according to an embodiment of the present invention. In the following description, descriptions of unnecessary embodiments that overlap with the descriptions of FIG. 1 to 16 described above will be omitted.

[0146] As illustrated in FIG. 17, the computing device (10000) may include at least one processor (11100), memory (11200), peripheral interface (11300), input / output subsystem (I / O subsystem) (11400), power circuit (11500), and communication circuit (11600). In this case, the computing device (10000) may correspond to a user terminal (A) connected to a haptic interface device or the aforementioned computing device (B).

[0147] The memory (11200) may include, for example, high-speed random access memory, a magnetic disk, SRAM, DRAM, ROM, flash memory, or non-volatile memory. The memory (11200) may include software modules, instruction sets, or various other data required for the operation of the computing device (10000).

[0148] At this time, access to memory (11200) from other components, such as the processor (11100) or peripheral device interface (11300), can be controlled by the processor (11100).

[0149] The peripheral device interface (11300) can connect input and / or output peripheral devices of the computing device (10000) to the processor (11100) and memory (11200). The processor (11100) can perform various functions for the computing device (10000) and process data by executing software modules or instruction sets stored in the memory (11200).

[0150] The input / output subsystem (11400) can connect various input / output peripherals to the peripheral interface (11300). For example, the input / output subsystem (11400) may include a controller for connecting peripherals such as a monitor, keyboard, mouse, printer, or, if necessary, a touchscreen or sensor to the peripheral interface (11300). According to another aspect, input / output peripherals may be connected to the peripheral interface (11300) without passing through the input / output subsystem (11400).

[0151] The power circuit (11500) can supply power to all or part of the components of the terminal. For example, the power circuit (11500) may include one or more power sources such as a power management system, a battery or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator, or any other components for power generation, management, and distribution.

[0152] The communication circuit (11600) can enable communication with another computing device using at least one external port.

[0153] Alternatively, as described above, the communication circuit (11600) may enable communication with other computing devices by including an RF circuit and transmitting and receiving an RF signal, also known as an electromagnetic signal.

[0154] The embodiment of FIG. 17 is merely an example of a computing device (10000), and the computing device (11000) may have some components shown in FIG. 17 omitted, additional components not shown in FIG. 17 added, or a configuration or arrangement that combines two or more components. For example, a computing device for a communication terminal in a mobile environment may include, in addition to the components shown in FIG. 17, a touchscreen or a sensor, etc., and the communication circuit (1160) may include a circuit for RF communication of various communication methods (WiFi, 3G, LTE, Bluetooth, NFC, Zigbee, etc.). The components that can be included in the computing device (10000) may be implemented as hardware, software, or a combination of both hardware and software, including one or more integrated circuits specialized for signal processing or applications.

[0155] Methods according to embodiments of the present invention may be implemented in the form of program instructions that can be executed through various computing devices and recorded on a computer-readable medium. In particular, the program according to the present embodiment may be configured as a PC-based program or an application dedicated to a mobile terminal. An application to which the present invention is applied may be installed on a user terminal through a file provided by a file distribution system. For example, the file distribution system may include a file transmission unit (not shown) that transmits the file upon a request from the user terminal.

[0156] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system.

[0157] Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0158] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed across networked computing devices and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0159] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory.

[0160] Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The above-mentioned hardware device may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0161] Although the embodiments have been described above with reference to limited embodiments and drawings, those skilled in the art can make various modifications and variations from the description above. For example, appropriate results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents. Therefore, other implementations, other embodiments, and equivalents to the claims below also fall within the scope of the claims.

Claims

Claim 1 A method for identifying finely biased text within a public corpus, implemented as a computing device comprising one or more processors and one or more memories storing instructions executable by said processors, comprising: a natural language sentence collection step of collecting natural language sentences from a text-based dataset including a public conversation dataset and a web corpus dataset; and a bias candidate dataset derivation step of deriving a bias candidate dataset including seed bias data expected to be socially and ethically biased and derived bias data extended from said seed bias data by comparing semantic vectors between words constituting said natural language sentences using a word embedding model in the collected natural language sentences. A method for identifying finely biased text in a public corpus, comprising: a bias determination step of determining the bias of a biased candidate dataset as one of a surface biased dataset, a finely biased dataset, and an unbiased dataset using a plurality of large-scale language models; wherein the bias determination step performs a bias determination in each of the plurality of large-scale language models for a biased candidate dataset to be judged, and if at least one of the plurality of large-scale language models determines the biased candidate dataset as a biased dataset, the biased candidate dataset is classified as a finely biased dataset. Claim 2 A method for identifying finely biased text within a public corpus, wherein, in claim 1, the bias determination step is characterized by using heterogeneous large-scale language models having different structures and training mechanisms as the plurality of large-scale language models. Claim 3 A method for identifying finely biased text within a public corpus, wherein, in claim 1, the bias judgment step is characterized by performing a bias judgment in each of the plurality of large-scale language models for a bias candidate dataset to be judged, and if all large-scale language models judge the bias candidate dataset to be a biased dataset, classifying the bias candidate dataset as a superficially biased dataset. Claim 4 A method for identifying finely biased text within a public corpus, wherein, in claim 1, the bias judgment step is characterized by performing a bias judgment in each of the plurality of large-scale language models for a bias candidate dataset to be judged, and classifying the bias candidate dataset as an unbiased dataset when all large-scale language models judge the bias candidate dataset to be an unbiased dataset. Claim 5 delete Claim 6 delete Claim 7 A method for identifying finely biased text within a public corpus, further comprising: a judgment criterion query step in which, when a single bias candidate dataset is classified as a finely biased dataset by performing the bias judgment step, a judgment criterion is queried to a large-scale language model that derives the classification result of the finely biased dataset; a response validity judgment step in which, when there is a response returned by the large-scale language model by performing the judgment criterion query step, the validity of the returned response is judged; and a training dataset storage step in which the finely biased dataset from which a valid response was derived in the response validity judgment step is classified as a training dataset for self-learning and stored in a training database. Claim 8 A method for identifying finely biased text within a public corpus, wherein, in claim 7, the step of determining response validity determines a valid response when an element including at least one of keywords and concepts including social and ethical issues is derived from the response returned by the large-scale language model. Claim 9 A method for identifying text with fine bias in a public corpus, characterized in that, in claim 7, after performing the bias judgment step, it further includes a self-learning step of providing learning data related to fine bias to the plurality of large-scale language models using a learning dataset stored in the learning database to enable self-learning. Claim 10 A method for identifying finely biased text and generating a corresponding response for identified finely biased text within a public corpus, implemented by a computing device comprising one or more processors and one or more memories storing instructions executable by said processors, comprising: a natural language sentence collection step of collecting natural language sentences from a text-based dataset including a public conversation dataset and a web corpus dataset; a bias candidate dataset derivation step of deriving a bias candidate dataset including seed bias data expected to be socially and ethically biased and derived bias data extended from said seed bias data by comparing semantic vectors between words constituting said natural language sentences using a word embedding model in the collected natural language sentences; a bias determination step of determining the bias in said bias candidate dataset as one of a surface bias dataset, a finely biased dataset, and an unbiased dataset using a plurality of large-scale language models; and a corresponding response generation step of generating a corresponding response for a natural language sentence identified as the surface bias dataset and said fine bias dataset by the bias determination step, wherein the corresponding response in which the bias is mitigated or removed is generated.A method for identifying finely biased text and generating a corresponding answer for identified finely biased text within a public corpus, comprising: a bias determination step receiving a bias determination result for a single bias candidate dataset from a plurality of large-scale language models, aggregating the received bias determination result to calculate a bias score for the single bias candidate dataset according to the principle of majority rule, defining a bias intensity based on a preset threshold bias score as a first bias level if the bias score is less than the threshold bias score, and as a second bias level if the bias score is greater than or equal to the threshold bias score; and a corresponding answer generation step generating a second corresponding answer composed of warning information warning of bias along with correction information correcting the bias when the bias intensity defined for the single bias candidate dataset is at the second bias level. Claim 11 A method for identifying finely biased text within a public corpus and generating a corresponding answer for identified finely biased text, wherein the step of generating a corresponding answer is characterized by generating a corresponding answer for a natural language sentence in which bias has been identified as a corresponding answer in a format including at least one of a summary format or a detailed description format. Claim 12 delete Claim 13 A method for identifying finely biased text within a public corpus and generating a corresponding response for the identified finely biased text, characterized in that, in the 10th paragraph, the confidence level for each of the above-mentioned multiple large-scale language models is defined by a pre-established confidence level management model, and in the bias score calculation step, the bias score is calculated by assigning the highest weight to the bias judgment result provided by the large-scale language model with the highest defined confidence level. Claim 14 A method for identifying finely biased text within a public corpus and generating a corresponding response for the identified finely biased text, wherein, in claim 10, the bias intensity definition step is defined as a first bias level when the bias score calculated for the one bias candidate dataset based on a preset threshold bias score is less than the threshold bias score, and is defined as a second bias level when the bias score calculated for the one bias candidate dataset is greater than or equal to the threshold bias score. Claim 15 A method for identifying finely biased text within a public corpus and generating a corresponding answer for identified finely biased text, wherein the corresponding answer generation step is characterized by generating a first corresponding answer, which is a corresponding answer composed of correction information that corrects the bias, when the bias strength defined for the first bias candidate dataset is the first bias level. Claim 16 delete Claim 17 An identification device for finely biased text within a public corpus, implemented as a computing device comprising one or more processors and one or more memories storing instructions executable by said processors, comprising: a natural language sentence collection unit for collecting natural language sentences from a text-based dataset including a public conversation dataset and a web corpus dataset; and a bias candidate dataset derivation unit for deriving a bias candidate dataset including seed bias data expected to be socially and ethically biased and derived bias data extended from said seed bias data by comparing semantic vectors between words constituting said natural language sentences using a word embedding model in the collected natural language sentences. A device for identifying finely biased text in a public corpus, comprising: a bias determination unit that determines the bias of a bias candidate dataset derived from a bias candidate dataset derivation unit using a plurality of large-scale language models as one of a surface bias dataset, a finely biased dataset, and an unbiased dataset; wherein the bias determination unit performs a bias determination in each of the plurality of large-scale language models for a bias candidate dataset to be judged, and if at least one of the plurality of large-scale language models determines the bias candidate dataset as a biased dataset, the bias candidate dataset is classified as a finely biased dataset. Claim 18 A device for identifying finely biased text within a public corpus and generating a corresponding response for the identified finely biased text, implemented as a computing device comprising one or more processors and one or more memories storing instructions executable by said processors, comprising: a natural language sentence collection unit for collecting natural language sentences from a text-based dataset including a public conversation dataset and a web corpus dataset; a bias candidate dataset derivation unit for deriving a bias candidate dataset including seed bias data expected to be socially and ethically biased and derived bias data extended from said seed bias data by comparing semantic vectors between words constituting said natural language sentences using a word embedding model in the collected natural language sentences; a bias determination unit for determining the bias of the bias candidate dataset derived by the bias candidate dataset derivation unit as one of a surface bias dataset, a finely biased dataset, and an unbiased dataset using a plurality of large-scale language models; and a corresponding response generation unit for generating a corresponding response for text data identified as the surface bias dataset and the finely biased dataset by the bias determination unit, wherein the corresponding response generates a response in which the bias is mitigated or removed.An apparatus for identifying finely biased text within a public corpus and generating a corresponding answer for identified finely biased text, comprising: a bias determination unit receiving a bias determination result for a single bias candidate dataset from a plurality of large-scale language models, aggregating the received bias determination result to calculate a bias score for the single bias candidate dataset according to the principle of majority rule, defining a bias intensity based on a preset threshold bias score as a first bias level if the bias score is less than the threshold bias score, and as a second bias level if the bias score is greater than or equal to the threshold bias score; and a corresponding answer generation unit generating a second corresponding answer composed of warning information warning of bias along with correction information correcting the bias when the bias intensity defined for the single bias candidate dataset is at the second bias level. Claim 19 In a computer-readable recording medium, the computer-readable recording medium stores instructions for a computing device to perform the following steps, the steps comprising: a natural language sentence collection step of collecting natural language sentences from a text-based dataset including a public conversation dataset and a web corpus dataset; and a bias candidate dataset derivation step of deriving a bias candidate dataset including seed bias data expected to be socially and ethically biased and derived bias data extended from the seed bias data by comparing semantic vectors between words constituting the natural language sentences using a word embedding model in the collected natural language sentences. A computer-readable recording medium comprising: a bias determination step of determining the bias of a bias candidate dataset as one of a surface bias dataset, a micro-bias dataset, and an unbiased dataset using a plurality of large-scale language models; wherein the bias determination step performs a bias determination in each of the plurality of large-scale language models for a bias candidate dataset to be judged, and if at least one of the plurality of large-scale language models determines the bias candidate dataset as a biased dataset, the bias candidate dataset is classified as a micro-bias dataset. Claim 20 In a computer-readable recording medium, the computer-readable recording medium stores instructions for a computing device to perform the following steps, the steps comprising: a natural language sentence collection step of collecting natural language sentences from a text-based dataset including a public conversation dataset and a web corpus dataset; a bias candidate dataset derivation step of deriving a bias candidate dataset including seed bias data expected to be socially and ethically biased and derived bias data extended from said seed bias data by comparing semantic vectors between words constituting said natural language sentences using a word embedding model in the collected natural language sentences; and a bias determination step of determining the bias in said bias candidate dataset as one of a surface bias dataset, a micro-bias dataset, and an unbiased dataset using a plurality of large-scale language models. A computer-readable recording medium comprising: a step for generating a response for natural language sentences identified as the surface bias dataset and the micro-bias dataset by the bias determination step, wherein the response generates a response in which the bias is mitigated or removed; wherein the bias determination step receives a bias determination result for a bias candidate dataset from a plurality of large-scale language models, aggregates the received bias determination result to calculate a bias score for the bias candidate dataset according to the principle of majority rule, and defines a bias intensity based on a preset threshold bias score as a first bias level if the bias score is less than the threshold bias score, and a second bias level if the bias score is greater than or equal to the threshold bias score; and wherein the response generating step generates a second response composed of warning information that warns of the bias along with correction information that corrects the bias when the bias intensity defined for the bias candidate dataset is at the second bias level.

Images