Information processing system, information processing method, and program

Abstract

To identify confidential information with high efficiency and accuracy by optimizing the amount of input information in a language model for determining the confidentiality of information.SOLUTION: An information processing system 100 includes an auxiliary storage device 203 that holds information of a sentence, and a processor 201 that calculates a calculation amount and a certainty factor expected at the time of confidentiality identification for each set of a specific sentence and a peripheral sentence on the basis of the information of the sentence, generates a peripheral information list indicating the peripheral sentence selected on the basis of a result of the calculation, and identifies whether the specific sentence is confidential information by inputting the specific sentence and the peripheral sentence to a language model on the basis of the peripheral information list.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention generally relates to information processing systems, information processing methods, and program technologies, and more specifically to a technology that optimizes the amount of input information in a language model that determines the confidentiality of information, thereby enabling the identification of confidential information with good efficiency and accuracy. [Background technology]

[0002] For example, confidential information helps a company secure its uniqueness in the market and gain a competitive advantage. Therefore, its owner must take appropriate measures to maintain the confidentiality of the confidential information. In order to properly manage confidential information, information managers in such companies set and manage confidentiality importance classifications, mainly on a document-by-document basis.

[0003] In the above management method, the information manager estimates the extent of damage that would occur if information were leaked based on the type of document and its overall content. Based on the results of this estimation, the information manager then determines the confidentiality importance classification for the document in question according to the degree of confidentiality protection required. However, in reality, there are many cases where confidential and non-confidential information is mixed within each document, making it difficult to accurately identify the confidential information. Furthermore, even if only a small portion of the information in a document is classified, the entire document will be classified.

[0004] In such cases, it becomes difficult to utilize the document, i.e., to utilize non-confidential information in the document. Therefore, a management method is required that divides the content referred to in the document into semantic units and manages confidential information in those semantic units. Note that, as a conventional technology for identifying confidential information, for example, a technology has been proposed (see Patent Document 1) that generates a publicly available document image without using confidential information based on a document image containing confidential information.

[0005] This technology relates to an information processing device that includes a means for acquiring character blocks of a first document image obtained by reading a document and character strings obtained by performing character recognition processing on each character block, a means for extracting character strings corresponding to named entities with predetermined attributes from the acquired character strings, and a means for generating document data including at least attribute information related to the attributes of the named entities corresponding to the extracted character strings and position information of the character blocks corresponding to the extracted character strings. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2023-087840 A Summary of the Invention [Problem to be solved by the invention]

[0007] On the other hand, there are also methods that identify confidential information by preparing registered keywords for confidential information in advance and detecting the registered keywords contained in the target document. However, these methods have the problem of not being able to identify confidential information that contains keywords other than the registered keywords. Furthermore, even in documents in which registered keywords appear, there are cases in which the information is non-confidential depending on the meaning of the sentence, and there is also the problem of insufficient accuracy.

[0008] On the other hand, one method for solving the above problem is to identify confidential information by taking into account the meaning of the document using natural language processing with a large-scale language model (hereinafter simply referred to as a "language model"), as shown in the above Patent Document 1. When a language model is used, it becomes possible to determine the confidentiality of a sentence to be identified in a document by taking into account the sentence in question.

[0009] For example, consider a case where a document contains two sentences, such as "Result AA was obtained. This is due to BB.", and the meaning of the sentence regarding "interpretation of the experimental results" needs to be identified as confidential information. In this case, it is difficult to identify the sentence "This is due to BB" on its own as a description regarding "interpretation of the experimental results." Therefore, in order to identify this sentence as confidential information, it is necessary to take into account the meaning of the sentence before (or after) it.

[0010] When using a language model to identify confidential information, in order to take into account the context surrounding the sentence to be identified, it is necessary to input not only the sentence to be identified but also the surrounding information into the language model. Of course, due to the characteristics of language models, if a large amount of surrounding information is input, improvement in identification accuracy can be expected. However, there is a concern that an increase in the amount of input information will increase the amount of calculation. Conversely, if the input of surrounding information is insufficient, there is a concern that identification accuracy will decrease.

[0011] The appropriate amount of peripheral information may also vary depending on the sentence to be classified. For example, in the above example, if the sentence to be classified is "The result AA was obtained," it can be easily identified as a description of experimental results without providing any peripheral information, so a small amount of peripheral information is sufficient. On the other hand, if the sentence to be classified is "It is BB," it cannot be identified as meaning "an interpretation of the experimental results" without providing any peripheral information, so a certain amount of peripheral information must be input.

[0012] As described above, there is a trade-off between identification accuracy and computational complexity, and the appropriate amount of peripheral information varies depending on various factors, making it difficult to determine. Therefore, the present invention has been made in consideration of the above-mentioned problems, and aims to provide a technology that optimizes the amount of input information in a language model that determines the confidentiality of information, thereby enabling identification of confidential information with good efficiency and accuracy. [Means for solving the problem]

[0013] The present application includes multiple means for solving the above-mentioned problems, examples of which are as follows: To solve the above-mentioned problems, an information processing system according to one aspect of the present invention comprises: a storage device that stores information on sentences present in at least a predetermined document; a process that, based on the information on the sentences, calculates an expected amount of calculation and a degree of certainty when inputting each set of a specific sentence to be determined for confidentiality and its surrounding sentences into a language model for confidentiality determination and identifying the confidentiality of the specific sentence; a process that generates a surrounding information list indicating the surrounding sentences selected based on the results of the calculation; and a processor that executes a process of inputting the specific sentence and the surrounding sentences into the language model based on the surrounding information list, thereby identifying whether the specific sentence is confidential information.

[0014] In addition, in order to solve the above problem, an information processing method according to one embodiment of the present invention is characterized in that an information processing system stores information on sentences present in at least a specified document in a storage device, and, based on the information on the sentences, calculates the expected amount of calculation and confidence level when inputting each set of a specific sentence and its surrounding sentences to be judged for confidentiality into a language model for confidentiality judgment to identify the confidentiality of the specific sentence; generates a surrounding information list indicating the surrounding sentences selected based on the results of the calculation; and, based on the surrounding information list, inputs the specific sentence and the surrounding sentences into the language model to identify whether the specific sentence is confidential information.

[0015] In addition, in order to solve the above problem, a program according to one aspect of the present invention is characterized in that it causes an information processing device that stores information on sentences present in at least a specified document in a storage device to execute the following steps: based on the information on the sentences, calculate the expected amount of calculation and confidence level when inputting each set of a specific sentence and its surrounding sentences to be judged for confidentiality into a language model for confidentiality judgment and identifying the confidentiality of the specific sentence; generating a surrounding information list indicating the surrounding sentences selected based on the results of the calculation; and identifying whether the specific sentence is confidential information by inputting the specific sentence and the surrounding sentences into the language model based on the surrounding information list. [Effects of the Invention]

[0016] According to the present invention, the amount of input information in a language model for determining the confidentiality of information is optimized, and confidential information can be identified with good efficiency and accuracy. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 2 is a diagram illustrating an example of a functional configuration of an information processing system according to a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing system according to a first embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of confidentiality classification information according to the first embodiment. [Figure 4] FIG. 10 is a diagram illustrating an example of policy information according to the first embodiment. [Figure 5] FIG. 2 is a diagram illustrating an example of document information according to the first embodiment. [Figure 6] FIG. 3 is a diagram illustrating an example of sentence information according to the first embodiment. [Figure 7] FIG. 2 is a diagram illustrating an example of confidential identification information according to the first embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of identification certainty information in the first embodiment. [Figure 9]FIG. 2 is a diagram illustrating an example of a flow of an information processing method according to the first embodiment. [Figure 10] FIG. 2 is a diagram illustrating an example of a flow of an information processing method according to the first embodiment. [Figure 11] FIG. 2 is a diagram illustrating an example of a flow of an information processing method according to the first embodiment. [Figure 12] FIG. 2 is a diagram illustrating an example of a flow of an information processing method according to the first embodiment. [Figure 13] FIG. 3 is a diagram illustrating an example of input to a language model in the first embodiment. [Figure 14] FIG. 10 is a diagram illustrating an example of a functional configuration of an information processing system according to a second embodiment. [Figure 15] FIG. 11 is a diagram illustrating an example of discrimination method evaluation value information in the second embodiment. [Figure 16] FIG. 10 is a diagram illustrating an example of a flow of an information processing method according to a second embodiment. [Figure 17] FIG. 10 is a diagram illustrating an example of a flow of an information processing method according to a second embodiment. [Figure 18] FIG. 10 is a diagram illustrating an example of output in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention should not be construed as being limited to the description of the embodiments shown below. Those skilled in the art will readily understand that the specific configuration can be changed without departing from the spirit or intent of the present invention.

[0019] In the configurations of the invention described below, identical or similar components or functions are denoted by the same reference numerals, and redundant explanations are omitted. The designations "first," "second," "third," etc. in this specification are used to identify components and do not necessarily limit the number or order. The position, size, shape, and range of each component shown in the drawings may not represent the actual position, size, shape, and range, etc., in order to facilitate understanding of the invention. Therefore, the present invention is not limited to the position, size, shape, and range, etc., disclosed in the drawings, etc.

[0020] In the following description, a communication device may be one or more communication interface devices, which may be one or more homogeneous communication interface devices (e.g., one or more NICs (Network Interface Cards)) or two or more heterogeneous communication interface devices (e.g., an NIC and an HBA (Host Bus Adapter)).

[0021] In the following description, a "memory" refers to one or more memory devices, which are an example of one or more storage devices. At least one of the memory devices may be a volatile memory device or a non-volatile memory device.

[0022] In the following description, a "storage device" may refer to one or more persistent storage devices, which are an example of one or more storage devices. A persistent storage device may typically be a non-volatile storage device, specifically, for example, a hard disk drive (HDD), a solid state drive (SSD), or a non-volatile memory express (NVMe) drive.

[0023] In the following description, a "CPU" refers to a computing device and may be one or more processor devices. The at least one processor device may typically be a microprocessor device such as a CPU (Central Processing Unit), but may also be another type of processor device such as a GPU (Graphics Processing Unit). The at least one processor device may be a single-core or multi-core. The at least one processor device may also be a processor core. The at least one processor device may also be a processor device in a broader sense, such as a hardware circuit that performs part or all of the processing (e.g., an FPGA (Field-Programmable Gate Array), a CPLD (Complex Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit)).

[0024] In the following description, information that provides an output in response to an input may be described using expressions such as "xxx table" or "xxx database." However, this information may be data of any structure (for example, structured data or unstructured data), or may be a learning model such as a neural network, genetic algorithm, or random forest that generates an output in response to an input. Therefore, "xxx table" or "xxx database" may be referred to as "xxx information." In the following description, the structure of each database or table is an example, and one database or table may be divided into two or more databases or tables, or all or part of two or more databases or tables may be one database or table.

[0025] In the following description, processing may be described using a "program" as the subject. However, since a program is executed by a CPU to perform a predetermined process using a storage device and / or an interface device, etc., as appropriate, the subject of the process may also be the CPU (or a device such as a controller having a processor). A program may be installed in a device such as a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable (e.g., non-transitory) recording medium. In the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0026] In addition, in the following description, when describing elements of the same type without distinguishing between them, common parts of the reference symbols may be used, and when describing elements of the same type with distinction between them, reference symbols or element identifiers may be used.

[0027] <Configuration of information processing system> FIG. 1 is a diagram illustrating a configuration example of an information processing system 100 according to a first embodiment. The information processing system 100 according to the first embodiment optimizes the amount of input information in a language model that determines the confidentiality of information, enabling efficient and accurate identification of confidential information. Specifically, the information processing system 100 optimizes peripheral information provided to the language model for each sentence to be identified, taking into consideration the trade-off between accuracy and computational complexity when using a language model to identify whether a sentence is classified as confidential information. Furthermore, the amount of peripheral information can be adjusted appropriately depending on the confidentiality importance, the information acquirer, and the confidentiality policy. According to the present disclosure, when managing confidential information for each sentence, it is possible to identify confidential information with a small amount of computational complexity and high identification accuracy. The following description will be given using confidential information handled in a company as an example of confidential information. [Example]

[0028] FIG. 1 is a diagram illustrating an example of a functional configuration of an information processing system 100 according to a first embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer 200 constituting the information processing system 100 according to the first embodiment. The information processing system 100 is configured from computers 200 as shown in FIG. 2. The number of computers 200 constituting the information processing system 100 is not limited. Such a computer 200 includes a processor 201, a main memory device 202, an auxiliary memory device 203, an input device 204, a display device 205, and a communication device 206. The hardware elements are connected to each other via a bus.

[0029] Of these, the input device 204 is a keyboard, mouse, or the like, and accepts input of data and instructions. The information manager operates the input device 204 to instruct the identification of confidential information regarding a specific document. Upon receiving the instruction, the information processing system 100 identifies the confidential information contained in the document. The display device 205 is a display, or the like, and displays setting information, processing results, and the like.

[0030] The main storage device 202 is a memory or the like that stores programs executed by the processor 201 and data used by the programs. The main storage device 202 is also used as a work area. The auxiliary storage device 203 is a non-volatile memory device such as an HDD (Hard Disk Drive) that permanently stores programs and information.

[0031] The auxiliary storage device 203 stores programs that realize the peripheral information list generation unit 101 and the confidential information identification unit 102. The auxiliary storage device 203 also has a memory unit 104, and stores confidential classification information 121, policy information 122, document information 123, sentence information 124, confidential identification information 125, and identification certainty information 126.

[0032] The processor 201 operates as a functional unit (module) that realizes a specific function by executing the above program. In the following description, when a process is described using a functional unit as the subject, it indicates that the processor 201 is executing a program that realizes the functional unit.

[0033] The communication device 206 communicates with a business system 50 that manages confidential information via a network 1. The communication device 206 may also be capable of communicating with a language model system 51 via the network 1. In this case, the information processing system 100 calls and uses the language model function of the present invention from the external language model system 51, and obtains the results. Of course, this is just one example, and does not limit the implementation or usage of the language model.

[0034] <Example of information configuration> Here, information held by the information processing system 100 will be described with reference to Figs. 3 to 8 in addition to Fig. 1. Fig. 3 is a diagram showing an example of the data structure of confidentiality classification information 121 in the first embodiment. The confidentiality classification information 121 is information for managing the classification of information treated as confidential. The confidentiality classification information 121 stores entries including a confidentiality classification ID 301, a confidentiality importance classification 302, an information acquirer classification 303, and a policy ID 304. One entry exists for one confidentiality classification. Note that the fields included in an entry are merely examples and are not limited to these. Any of the above-mentioned fields may not be included, or other fields may be included.

[0035] The confidentiality classification ID 301 is a field for storing the ID of the confidentiality classification, and is the same field as the confidentiality classification ID 701 and confidentiality classification ID 801 described later. The confidentiality importance classification 302 is a field for storing information indicating the importance of confidentiality (for example, public, confidential, secret, top secret, etc.), and is the same field as the confidentiality importance classification 502 described later. The information acquirer classification 303 is a field for storing the classification related to the person acquiring the information (for example, an external customer or an internal department, etc.). The policy ID 304 is a field for storing the ID of the confidentiality policy, which is the meaning of what is considered confidential information, and is the same field as the policy ID 401 described later. The confidentiality classification information 121 may be set in advance or acquired from outside.

[0036] FIG. 4 is a diagram showing an example of the data structure of the policy information 122 in the first embodiment. This policy information 122 is information for managing confidentiality policies. The policy information 122 stores entries each including a policy ID 401, a policy content 402, a policy keyword 403, and an embedded expression vector 404. One entry exists for one confidentiality policy. Note that the fields included in the entry are merely examples and are not limited to these. Any of the above-mentioned fields may not be included, or other fields may be included.

[0037] Of these, policy ID 401 is a field for storing the ID of the confidentiality policy. Policy content 402 is a field for storing the content of the confidentiality policy in a free description format. Policy keyword 403 is a field for storing a keyword that matches the content of the confidentiality policy. Embedded expression vector 404 is a field for storing a value obtained by converting policy content 402 into an embedded expression vector. Policy information 122 may be set in advance or obtained from an external source.

[0038] FIG. 5 is a diagram showing an example of the data structure of document information 123 in the first embodiment. This document information 123 is information for an information manager to manage documents. The document information 123 stores entries including a document ID 501, a confidentiality importance classification 502, and a document name 503. One entry exists for one document. Note that the fields included in the entry are merely examples and are not limited to these. Any of the above-mentioned fields may not be included, or other fields may be included.

[0039] Of these, document ID 501 is a field that stores the ID of a document, and is the same field as document ID 602, which will be described later. Confidentiality Importance Classification 502 is a field that stores information indicating the importance of confidentiality. Document Name 503 is a field that stores the name of a document. Document information 123 may be set in advance or obtained from an external source.

[0040] FIG. 6 is a diagram showing an example of the data structure of sentence information 124 in the first embodiment. This sentence information 124 is information for managing sentences in a document. The sentence information 124 stores entries including a sentence ID 601, a document ID 602, text 603, a token count 604, and an embedded expression vector 605. One entry exists for each combination of a sentence ID and a document ID. Note that the fields included in an entry are merely examples and are not limited to these. Any of the above-mentioned fields may be omitted, or other fields may be included.

[0041] Of these, sentence ID 601 is a field that stores the ID of a specific sentence in a document, and is the same field as sentence ID 702 and sentence ID 802, which will be described later. Document ID 602 ​​is a field that stores the ID of the document to which the sentence stored in the sentence ID belongs. Text 603 is a field that stores the content of the sentence written in free text format.

[0042] The number of tokens 604 is a field that stores the number of tokens, which are basic units of text obtained by dividing a character string into meaningful chunks. The number of tokens represents the length of the text, and as will be described later, the amount of calculation for the language model is estimated based on this number. The embedded expression vector 605 is a field that stores a value obtained by converting the text 603 written in a free description format into an embedded expression vector. The sentence information 124 may be set in advance or obtained externally.

[0043] FIG. 7 is a diagram showing an example of the data structure of the confidential identification information 125 in the first embodiment. This confidential identification information 125 is information for managing an identification result regarding whether a certain sentence corresponds to confidential information when considered from the perspective of a specific confidential classification. The confidential identification information 125 stores entries including a confidential classification ID 701, a sentence ID 702, and an identification result 703. One entry exists for each combination of a confidential classification ID and a sentence ID. Note that the fields included in an entry are merely examples and are not limited to these. Any of the above-mentioned fields may be omitted, or other fields may be included.

[0044] Of these, confidentiality classification ID 701 is a field that stores the ID of a confidentiality classification. Sentence ID 702 is a field that stores the ID of a specific sentence. Identification result 703 is a field that stores whether a certain sentence corresponds to confidential information when considered from the perspective of a specific confidentiality classification. In this embodiment, either "1" indicating that it corresponds to confidential information or "0" indicating that it does not correspond to confidential information is set. Confidentiality identification information 125 may be set in advance or may be obtained from outside.

[0045] 8 is a diagram showing an example of the data structure of the identification certainty information 126 in the first embodiment. The identification certainty information 126 is information for managing the identification certainty when a certain sentence is identified as classified information from the perspective of a specific classified classification, and is information for managing a list of peripheral information (peripheral sentences) generated by the information processing system 100. There may be cases where multiple peripheral information lists are generated for one combination of sentence and classified classification.

[0046] Of these, the identification certainty information 136 stores entries including a confidentiality classification ID 801, a sentence ID 802, a peripheral information list 803, an identification result 804, an identification certainty 805, and a peripheral information list evaluation value 806. One entry exists for each combination of a confidentiality classification ID, a sentence ID, and a peripheral information list. Note that the fields included in an entry are merely examples and are not limited to these. Any of the above-mentioned fields may not be included, and other fields may also be included.

[0047] Of these, the confidential classification ID 701 is a field that stores the ID of the confidential classification. The sentence ID 702 is a field that stores the ID of a specific sentence. The peripheral information list 803 is a field that stores, in the form of a list of sentence IDs, sentences of peripheral information that were input to the language model when identifying confidential information. The identification result 804 is a field that stores the identification result of confidential information output by the language model. The identification certainty 805 is a field that stores the identification certainty regarding the identification of confidential information output by the language model. The peripheral information list evaluation value 806 is a field that stores an evaluation value calculated by the information processing system 100 based on the sentence to be identified and the peripheral information list. The identification certainty information 126 may be set in advance or obtained from outside.

[0048] <Information processing method> 9 is a flowchart illustrating processing executed by the information processing system 100 according to the first embodiment. When the information processing system 100 according to the first embodiment receives, from the input device 204 or the like, an instruction requesting identification of sentences containing confidential information in a specific document, the information processing system 100 starts the processing described below.

[0049] First, the peripheral information list generation unit 101 of the information processing system 100 receives the sentence to be identified (document information 123 and sentence information 124), information on its peripheral sentences (sentence information 124), and information on the confidentiality policy (policy information 122) in the above-mentioned instructions, etc., and generates a peripheral information list to be input to a language model (for example, used via the language model system 51) used to identify whether the sentence to be identified (specific sentence) contains confidential information. The peripheral information list generation unit 101 includes an identification certainty prediction unit 111.

[0050] Next, the identification certainty prediction unit 111 of the information processing system 100 calculates a predicted value of the identification certainty based on the confidentiality policy, the sentence to be identified, and the peripheral information list. Also, the confidential information identification unit 102 of the information processing system 100 receives information related to the sentence to be identified, the peripheral information list generated by the peripheral information list generation unit 101, and the confidentiality policy, and identifies whether the target sentence contains confidential information.

[0051] Here, an overview of the processing executed by the information processing system 100 of the first embodiment will be described. It is assumed that the document ID of the document to be processed is prepared in advance in the information processing system 100. First, the information processing system 100 determines a confidentiality classification ID for the document to be identified, which is composed of a confidentiality importance classification, an information acquirer classification, and a confidentiality policy (step S100). The confidentiality importance classification is obtained by acquiring the confidentiality importance classification 502 of the document whose document ID matches from the document information 123. The information acquirer classification and confidentiality policy are set by the information administrator according to the purpose of protecting confidential information. Although it is possible to select multiple information acquirer classifications and confidentiality policies, in this embodiment, one of each is selected and processing is executed. In this way, the information processing system 100 selects one of the classification IDs listed in the confidentiality classification information 121.

[0052] Next, the information processing system 100 sets the classification confidence threshold conf thres (Step S200). thres The higher the threshold conf, the stricter the criteria for identifying confidential information. thres If the threshold is not higher than , the processing is set so that confidential information is not identified using the language model. Any method for determining the threshold is allowed.

[0053] Next, the information processing system 100 selects a sentence included in the target document (step S300). Such a sentence may be selected from the sentence information 124, any sentence whose document ID 602 ​​matches the document ID of the target document.

[0054] Next, the surrounding information list generating unit 101 of the information processing system 100 executes a surrounding information list generating process (step S400). In this surrounding information list generating process, the identification certainty factor predicting unit 111 executes an identification certainty factor predicting process (step S500).

[0055] Next, the confidential information identification unit 102 of the information processing system 100 executes the confidential information identification process (step S600). The information processing system 100 also determines whether the processes from step S400 to step S600 have been executed for all sentences included in the specified document ID (step S700).

[0056] If the result of the above determination is that processing has not been performed on all sentences (step S700: No), the information processing system 100 returns to step S300 and similarly executes each step from step S300 onwards.On the other hand, if the result of the above determination is that processing has been performed on all sentences (step S700: Yes), the information processing system 100 completes the classification processing for the document that is the target of this processing.

[0057] (Nearby information list generation process) Next, a description will be given of the details of each process executed by the information processing system 100. Fig. 10 is a flowchart illustrating a peripheral information list generation process executed by the information processing system 100 of the first embodiment. Fig. 11 is a flowchart illustrating an identification certainty factor prediction process executed by the information processing system 100 of the first embodiment.

[0058] In this case, first, the peripheral information list generation unit 101 acquires (step S401) a confidentiality policy from the confidentiality classification ID selected in processing step S100 executed by the information processing system 100, and the sentence selected in step S300. The confidentiality policy is acquired by acquiring a policy ID 304 from an entry having the same confidentiality classification ID 301 as the confidentiality classification ID selected from the confidentiality classification information 121, and acquiring a policy content 402 and an embedded expression vector 404 from an entry having the same policy ID 401 as the policy ID acquired from the policy information 122.

[0059] Also, for the sake of explanation, if the selected sentence is sentence M, the peripheral information list generation unit 101 obtains information about the sentence having the sentence ID 601 corresponding to sentence M from the sentence information 124, namely, the number of tokens 604 and the embedded expression vector 605.

[0060] Next, the peripheral information list generation unit 101 acquires information about all sentences that are candidates for the peripheral information list (step S402). In this embodiment, as an example of a candidate for the peripheral information list, the sentences N before and N after the selected sentence will be taken as an example. For the sake of explanation, the number of sentences in the entire document is assumed to be L, and M is assumed to be greater than N and greater than LN.

[0061] Specifically, for all i taking integer values ​​between -N and +N, the peripheral information list generation unit 101 acquires information about sentences having sentence ID 601 corresponding to sentence M+i from sentence information 124, i.e., the number of tokens 604 and the embedded expression vector 605. Furthermore, if M is smaller than N, the acquired value is set to "0" for all i for which M+i is smaller than 0. If M is larger than LN, the acquired value is set to "0" for all i for which M+i is greater than N.

[0062] In the embodiment, for simplicity, the peripheral information for sentence M is set to sentence M+i, but the method for setting the peripheral information is not limited to this. For example, the summary information of the paragraph to which sentence M belongs, the summary information of the paragraphs before and after it, the summary information of the document to which the sentence belongs, and the document name 503 can also be set as peripheral information. In this case, this information can be stored in advance in the sentence information 124 and processed in the same way as the peripheral information sentence M+i.

[0063] Next, the peripheral information list generation unit 101 sets a peripheral information list for the selected sentence (step S403). In this case, specifically, the peripheral information list generation unit 101 sets a set of binary variables X_i as information representing the peripheral information list. The binary variable X_i is a variable that is set to "1" if sentence M+i, which is one of the peripheral information of sentence M, is to be included in the peripheral information list, and is set to "0" if it is not. Note that the initial peripheral information list may be set randomly or according to any rule. Furthermore, if M is smaller than N, X_i may be set to "0" for all i such that M+i is smaller than 0. If M is greater than LN, X_i may be set to "0" for all i such that M+i is greater than N. Furthermore, if there is no pre-update value for the peripheral information list evaluation value, this is set to any value.

[0064] Next, the neighborhood information list generation unit 101 calculates an estimated calculation amount based on the number of tokens in the acquired neighborhood information list (step S404). While any method for calculating the estimated calculation amount may be used, in the present disclosure, as an example, the estimated calculation amount is calculated by substituting it into Equation 1.

[0065]

number

[0066] Here, C is the estimated amount of calculation, c1 is a proportionality constant for calculating the amount of calculation from the number of tokens, and T_i is the number of tokens in sentence M+i.

[0067] The peripheral information list generation unit 101 instructs the identification certainty prediction unit 111 to execute an identification certainty prediction process (step S405). At this time, the peripheral information list generation unit 101 inputs information about the confidentiality policy, the target sentence M, and all sentences from sentences M-N to sentences M+N, which are peripheral information list candidates, to the identification certainty prediction unit 111. Here, the identification certainty prediction process will be described with reference to FIG. 11.

[0068] The classification certainty prediction unit 111 obtains the embedded expression vectors of the target sentence M and its neighboring information list candidate sentence M+i, and calculates the similarity between sentence M and sentence M+i (step S501). While any method for calculating the similarity may be used, in the present disclosure, as an example, the cosine similarity is calculated by substituting the estimated calculation amount into the following formula 2.

[0069]

number

[0070] Here, S i is the cosine similarity between sentence M and sentence M+i, B M is the embedding vector of sentence M, |X| is the size of vector X,<X·Y> represents the dot product of vector X and vector Y.

[0071] The identification certainty prediction unit 111 obtains a predicted value of the identification certainty using a prepared prediction model (step S502). As an example of the form of the prediction model, in the present disclosure, an embedded representation B P , the embedding representation B of the target sentence M M , the inner product value of the target sentence M and its embedding representation S i , and X i is input and a predicted value of the identification confidence is output. Any method for calculating the identification confidence can be used, but in this disclosure, the probability that the output corresponds to confidential information when confidential information is identified using a language model is set as P, and the identification confidence is calculated by substituting this into the following equation 3.

[0072]

number

[0073] Here, conf represents the identification certainty. The closer the probability P of guessing that the information is confidential is to 0 or 1, the larger the value of the above formula 3 becomes. Conversely, if P is close to 0.5 and the probability that the information is confidential is roughly the same as the probability that it is not confidential, the value of formula 3 becomes smaller. Therefore, formula 3 represents the certainty regarding the identification of confidential information.

[0074] The prediction model to be prepared in advance can be constructed using any model. In this disclosure, construction of a prediction model using a neural network model will be described as an example. The embedded representation B of the confidentiality policy to be targeted by this input variable is P can be obtained from the policy information 122, and the embedded representation B of sentence M M , the similarity between the embedding representations of the target sentence M and the surrounding information sentence M+i is S i , and X i can be obtained from the identification certainty information 126 and the sentence information 124. For one entry of the identification certainty information 126, the embedded representation of the target sentence and its peripheral information list candidate can be obtained from the sentence information 124, and S i It is possible to calculate X i can be obtained from the peripheral information list. To obtain the identification confidence, which is the output of the model, data related to the identification result can be used as the objective variable. This can be obtained by obtaining the identification result 703 related to the target sentence from the confidential identification information 125. By constructing a learning model using the pairs of input variables and output variables prepared in the above procedure as learning data, it is possible to construct a model that can obtain a predicted value of the identification confidence.

[0075] In addition, in the embodiment, for simplicity, the input variables of the prediction model are the embedded expression of the confidentiality policy, the embedded expression of the target sentence, the similarity between the embedded expressions of the target sentence and the sentences of the peripheral information, and the peripheral information list, but the way in which the input variables are set is not limited to this. For example, in addition to these, the number of occurrences of important keywords and the number of privacy identifiers included in the target sentence and the peripheral information can be added as input variables.

[0076] The important keyword can be, for example, the policy keyword 403 in the policy information 122. The number of occurrences of important keywords can be obtained by calculating the number of occurrences of important keywords contained in the target sentence and peripheral information based on a keyword list prepared in advance. The privacy information identifier can be obtained by calculating the number of occurrences of important keywords contained in the target sentence and peripheral information based on a privacy information identifier detection device prepared in advance.

[0077] In addition, the above-mentioned privacy information identifier detection device can be obtained, for example, using a language model that has been pre-trained to detect privacy information identifiers, separate from the language model described in this disclosure, and this can be used to obtain privacy information identifiers in natural language.

[0078] The identification certainty factor prediction unit 111 outputs the predicted value of the identification certainty factor to the peripheral information list generation unit 101, and ends the identification certainty factor prediction process.

[0079] Returning now to the explanation of Fig. 10, the surrounding information list generation unit 101 calculates a surrounding information list evaluation value (step S406). Specifically, the surrounding information list generation unit 101 calculates the surrounding information list evaluation value using the identification certainty (the "identification certainty" of the present invention) and the estimated calculation amount (the "calculation amount" of the present invention). In the present disclosure, an example of this evaluation value is calculated by substituting it into the following formula 4.

[0080]

number

[0081] Here, Eval list is the evaluation value of the neighboring information list, A is a sufficiently large constant, conf thres indicates the threshold given to the classification confidence. U(X) is a step function, which is 1 when X>0 and 0 when X<0. Eval list The smaller the value, the higher the evaluation.thres If it is higher than , the second term will be the minimum value of 0, and Eval list Also, the smaller the estimated computational complexity C, the faster Eval list becomes smaller.

[0082] The peripheral information list evaluation value in the first embodiment is an index that takes into consideration the estimated amount of calculation and constraints on the identification certainty.

[0083] In the first embodiment, the peripheral information list evaluation value was determined by substituting it into the above formula 4, but the method of determining the evaluation value is not limited to this. For example, formula 4 is intended to impose a constraint on the magnitude of the identification certainty conf, but it is also possible to store the identification certainty conf previously calculated by the peripheral information list generation unit 101 as the pre-update value of the identification certainty, and to define the evaluation formula so as to impose a constraint on the amount of change between the post-update value and the pre-update value of the identification certainty, using the calculated identification certainty as the post-update value of the identification certainty.

[0084] Next, the surrounding information list generation unit 101 determines whether the surrounding information list evaluation value has been sufficiently improved (step S407). Specifically, the updated value of the surrounding information list evaluation value acquired in step S406 is compared with the pre-update value of the surrounding information list evaluation value. As an example, if the updated value of the surrounding information list evaluation value is smaller, the value of the surrounding information list evaluation value is smaller than a set constant A, and the difference between the updated value and the pre-update value is smaller than a predetermined threshold, it can be determined that the surrounding information list evaluation value has been sufficiently improved.

[0085] If this condition is met (step S407: Yes), the surrounding information list generating unit 101 sets the binary variable X i The set of the above is acquired as a neighborhood information list, and the process ends (step S408). On the other hand, if this condition is not met (step S407: No), the neighborhood information list generation unit 101 determines whether the process of generating the neighborhood information list, S403, has been performed a certain number of times or more (step S409).

[0086] If the result of the above determination is that the generation process has not been performed a certain number of times or more (step S409: No), the surrounding information list generation unit 101 returns to S403. The surrounding information list evaluation value is substituted as the pre-update value of the surrounding information list evaluation value. In this step S403, the surrounding information list is generated so as to search for the surrounding information list with the smallest surrounding information list evaluation value. In the first embodiment, for example, a heuristic optimization method such as a genetic algorithm can be used.

[0087] On the other hand, if the result of the above judgment is that the above generation process has been performed a certain number of times or more (step S409: Yes), the surrounding information list generation unit 101 determines that it was difficult to appropriately define the surrounding information list, and terminates the process without obtaining the surrounding information list (step S410).

[0088] As described above, the peripheral information list generation unit 101 can appropriately acquire a peripheral information list by taking into consideration the trade-off between the amount of calculation required to identify whether a target sentence contains confidential information and the certainty of the identification. The identification certainty prediction unit 111 can acquire an identification certainty used when the peripheral information list generation unit 101 calculates a peripheral information list evaluation value.

[0089] (Confidential information identification processing) 12 is a flowchart illustrating the confidential information identification process executed by the information processing system 100 of the first embodiment. The peripheral information list and peripheral information list evaluation value output from the peripheral information list generation unit 101 are input to the confidential information identification unit 102. The information processing system 100 also inputs information regarding the confidentiality policy, the target sentence, and the set of sentences in the peripheral information list to the confidential information identification unit 102.

[0090] Here, the confidential information identification unit 102 checks whether it can acquire a peripheral information list (step S601). If the result of the above check is that it is not possible to acquire a peripheral information list (step S601: No), the confidential information identification unit 102 acquires a manual identification result for the confidential information (step S602). Since the peripheral information list generation unit 101 was unable to generate a peripheral information list that satisfies the conditions regarding the amount of calculation and the identification certainty, and therefore is unable to acquire a peripheral information list, it is difficult to identify whether the target sentence contains confidential information using a language model, so the identification result is acquired manually. However, in order to continue mechanical processing, it is not necessarily necessary to require manual input of the identification result in step S602.

[0091] The confidential information identification unit 102 also inputs the confidentiality policy, the target sentence, and the manual identification result as one entry in the identification certainty information 126 (step S603). At this time, the set of sentences in the peripheral information list, the peripheral information list evaluation value, and the identification certainty are all set as empty sets. However, if the manual identification result was not input in step S602, the manual identification result is also input as an empty set.

[0092] On the other hand, if the result of the above check is that the peripheral information list can be obtained (step S601: Yes), the confidential information identification unit 102 inputs text information of the confidentiality policy, the target sentence, and the set of sentences in the peripheral information list into the language model to obtain the identification result regarding the confidential information in the language model (step S604). Figure 13 shows an example 1300 of input to the language model.

[0093] The confidential information identification unit 102 also acquires the output of the language model, i.e., the identification result and identification certainty for the confidential information (step S605). The confidential information identification unit 102 inputs the confidentiality policy, the target sentence, the set of sentences in the peripheral information list, the peripheral information list evaluation value, the identification result calculated in step S605, and the identification certainty as one entry in the identification certainty information 126 (step S606).

[0094] As described above, the confidential information identifying unit 102 can identify whether the target sentence contains confidential information by providing the confidentiality policy, the target sentence, and a set of sentences in the peripheral information list.

[0095] According to the first embodiment, it is possible to optimize peripheral information provided to a language model for each sentence to be classified, taking into consideration the trade-off between classification accuracy and calculation amount. However, the application of the first embodiment is not limited to supporting the classification of confidential information. That is, although the first embodiment shows a process for identifying whether a specific content corresponding to confidential information is included, it can be used as a process for identifying whether a specific content other than confidential information is included.

[0096] Here, consider a request for proposal (RFP) as an example of a document, and "availability" as an example of specific content. This embodiment also aims to identify whether a sentence in the document mentions "availability" while taking into account the context of the sentence. Furthermore, by calculating the number of sentences mentioning "availability" in the document and determining whether the number exceeds a predetermined threshold, it is possible to determine whether a certain RFP adequately describes "availability." [Example]

[0097] Next, the threshold conf of the classification certainty determined by the information processing system 100 of the first embodiment is thres The second embodiment will be described focusing on the differences from the first embodiment, such as the addition of a process for determining an appropriate value for the first embodiment.

[0098] 14 is a diagram illustrating an example of a functional configuration of an information processing system 100 according to a second embodiment. The hardware configuration of the computer constituting the information processing system 100 according to the second embodiment is the same as that according to the first embodiment. The second embodiment differs in that the auxiliary storage device 203 stores the distinction method evaluation value information 127. The second embodiment differs in that the auxiliary storage device 203 stores a program that realizes the distinction method evaluation unit 103. The second embodiment differs in the outline of the processing executed by the information processing system 100.

[0099] 15 is a diagram illustrating an example of the data structure of the identification method evaluation value information 127 according to the first embodiment. The identification method evaluation value information 127 is information for managing evaluation values ​​related to identification methods for confidential information, and is generated by the information processing system 100. The identification method evaluation value information 127 stores entries including a confidentiality classification ID 901, an identification confidence threshold 902, an identification accuracy 903, an estimated calculation amount 904, a manual identification ratio 905, and an identification method evaluation value 906.

[0100] There is one entry for each combination of confidentiality classification ID and identification certainty threshold. Note that the fields included in the entry are merely examples and are not limited to these. An entry may not include any of the above fields, or may include other fields.

[0101] Among these, the confidentiality classification ID 901 is a field for storing the ID of the confidentiality classification, and is the same field as the confidentiality classification ID 301. The identification certainty threshold 902 is a field for storing the identification certainty threshold, which is an input variable of the information processing system 100. The identification accuracy 903 is a field for storing the identification accuracy calculated by the information processing system 100 based on the confidentiality classification ID and the identification certainty threshold. The estimated calculation amount 904 is a field for storing the estimated calculation amount calculated by the information processing system 100 based on the confidentiality classification ID and the identification certainty threshold. The manual identification ratio 905 is a field for storing the manual identification ratio calculated by the information processing system 100 based on the confidentiality classification ID and the identification certainty threshold. The identification method evaluation value 906 is a field for storing an evaluation value related to the identification method calculated by the information processing system 100 based on the confidentiality classification ID and the identification certainty threshold. The identification method evaluation value information 127 may be set in advance or obtained from an external source.

[0102] 16 is a flowchart illustrating a process executed by the information processing system 100 according to the second embodiment. In the first embodiment, the information processing system 100 starts the process when it receives an instruction to identify sentences containing confidential information in a specific document. In the second embodiment, the information processing system 100 starts the process when it receives an instruction to identify sentences containing confidential information in a specific document. In the first embodiment, the information processing system 100 starts the process when it receives an instruction to identify sentences containing confidential information in a specific document. thres When an instruction is received to determine an appropriate value according to the confidentiality importance classification, information acquirer classification, manual work ratio, etc., the processing is started.

[0103] In the second embodiment, a classification method evaluation unit 103 is used in addition to the peripheral information list generation unit 101 and the confidential information identification unit 102 used in the first embodiment. The classification method evaluation unit 103 calculates a classification method evaluation value based on the classification accuracy, estimated calculation amount, and manual classification ratio, which are values ​​calculated by the information processing system 100. By evaluating the classification device using the calculated evaluation value, a classification certainty threshold conf determined by the information processing system 100 is obtained. thres shall be appropriately determined.

[0104] An outline of the process executed by the information processing system 100 of the second embodiment will be described. The processes of steps S100 to S600 in the second embodiment are the same as those in the first embodiment. It is determined whether the processes of steps S400 to S600 have been executed for all sentences included in the specified document ID (step S700).

[0105] If the result of the above determination is that the processing has not been performed on all sentences (step S700: No), the classification method evaluation unit 103 returns to step S300 and similarly executes the processing from step S300 onwards. On the other hand, if the result of the above determination is that the processing has been performed on all sentences (step S700: Yes), the classification method evaluation unit 103 executes classification method evaluation processing (step S800).

[0106] Furthermore, the classification method evaluation unit 103 determines whether the classification method evaluation value has been sufficiently improved (step S900). If the result of the above determination is that the classification method evaluation value has not been sufficiently improved (step S900: No), the classification method evaluation unit 103 returns to step S200 and executes the subsequent processes in the same manner. Note that in step S200, the classification method evaluation unit 103 determines the classification confidence threshold conf that minimizes the classification method evaluation value. thres The threshold conf of the classification confidence is used to search for thres In the second embodiment, for example, an optimization method such as Bayesian optimization is used to generate a threshold conf thres can be explored.

[0107] If the result of the above determination is that the classification method evaluation value has been sufficiently improved (step S900: Yes), the classification method evaluation unit 103 completes the process and sets the classification certainty threshold conf thres Determine.

[0108] 17 is a flowchart illustrating the identification method evaluation process executed by the information processing system 100 of the embodiment 2. In this flow, when the information processing system 100 executes the confidential information identification process, all entries stored in the identification certainty information 126 are input to the identification method evaluation unit 103.

[0109] In this case, the identification method evaluation unit 103 calculates the identification accuracy (step S801). To do this, the identification result 804 is obtained for each entry of the identification certainty information 126 input to the identification method evaluation unit 103, and an entry of the confidential identification information 125 having the same sentence ID and confidentiality classification ID is searched for to obtain its identification result 703. In the second embodiment, as an example of the identification accuracy, it is calculated by substituting into the following formula 5, which indicates the accuracy rate, which is the rate at which both identification results match.

[0110]

number

[0111] Here, Accuracy represents the accuracy rate, and L represents the total number of entries of the identification certainty information 126 input to the identification method evaluation unit 103. True_j represents "1" if, for the j-th entry of the identification certainty information 126 input to the identification method evaluation unit 103, the identification result of the identification certainty information 126 matches the identification result of the confidential identification information 125 having the same sentence ID and confidentiality classification ID, and represents "0" if they do not match.

[0112] Furthermore, the classification method evaluation unit 103 calculates an estimated computational amount (step S802). To this end, for each entry of the classification certainty information 126 input to the classification method evaluation unit 103, the sentence ID of each entry and the number of tokens of the sentence ID in the peripheral information list are acquired from the sentence information 124. Any method can be used to calculate the estimated computational amount based on the number of tokens, but in the present disclosure, as an example, the estimated computational amount is calculated by substituting it into the following formula 6.

[0113]

number

[0114] where Calc is the estimated computational complexity, c1 is the proportional constant for calculating the computational complexity from the number of tokens, and T j+i represents the number of tokens in sentence j+i. Xij is a binary variable that indicates whether or not the jth entry sentence of the classification certainty information 126 input to the classification method evaluation unit 103 contains peripheral information i. For the sake of explanation, the number of sentences in the entire document is assumed to be L, so if j+i is smaller than 0 or if j+i is greater than L, the number of tokens is set to "0."

[0115] Next, the classification method evaluation unit 103 calculates the manual classification ratio (step S803). In the second embodiment, the manual classification ratio can be calculated by counting the ratio of the neighbor information list being an empty set for each entry of the classification certainty information 126 input to the classification method evaluation unit 103. In the present disclosure, the manual classification ratio can be calculated by substituting into the following formula 7.

[0116]

number

[0117] Here, Work_rate represents the manual classification rate, and Empty_j represents "1" if the peripheral information list of the jth entry of the classification certainty information 126 input to the classification method evaluation unit 103 is an empty set, and "0" if it is not an empty set.

[0118] The classification method evaluation unit 103 calculates a classification method evaluation value taking into consideration the classification accuracy rate, the estimated calculation amount, and the manual classification ratio (step S804). In the present disclosure, as an example, the classification method evaluation value is calculated by substituting it into the following formula 8.

[0119]

number

[0120] Here, Eval method represents the classification method evaluation value. α, β, and γ are proportional constants assigned to the accuracy rate Accuracy, the estimated calculation amount Calc, and the manual classification rate Work_rate, respectively. method is the classification confidence threshold confthres It is a value that changes depending on the threshold conf of the classification confidence. thres When the above is set, the accuracy rate Accuracy, the estimated calculation amount Calc, and the manual classification rate Work_rate are determined, and these are the classification confidence threshold conf thres This is because it changes depending on Eval method means that the smaller the value, the higher the evaluation. The evaluation value becomes smaller when the accuracy rate Accuracy becomes higher, the estimated calculation amount Calc becomes smaller, or the manual classification rate Work_rate becomes smaller. The classification method evaluation value of the second embodiment is an index that takes into account the classification accuracy, the estimated calculation amount, and the manual classification rate.

[0121] 18 is a diagram showing an example of a screen presented by the information processing system 100 of the second embodiment. The screen 1400 of the second embodiment is also displayed in the same manner as in the first embodiment. However, unlike the screen in the second embodiment, the screen displayed in the first embodiment does not include an evaluation value output field 1402. Specifically, the screen 1400 in the second embodiment displays a peripheral information list evaluation value, a peripheral information list, a classification method evaluation value, and a classification confidence threshold.

[0122] The information processing system 100 of the first and second embodiments can be used in a variety of ways. For example, by using the information processing system 100, it is possible to preliminarily screen whether a document created within an organization contains confidential information before the document is made public or before an email is sent to an information acquirer with a different management classification, and to present any portions that require correction. Furthermore, by using the information processing system 100, it is possible to identify non-confidential information that has previously been treated as confidential information. By appropriately managing and utilizing this information according to the information acquirer, it is possible to improve the accuracy of interactive AI while protecting confidentiality, for example, and to more efficiently support investigation work.

[0123] The present invention is not limited to the above-described embodiments, but includes various modifications. For example, the above-described embodiments are provided to explain the present invention in detail, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, some of the configurations of each embodiment can be added to, deleted from, or replaced with other configurations.

[0124] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The present invention can also be realized by software program code that implements the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a computer, and the processor 201 of the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium implements the functions of the above-described embodiments, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media for providing such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, solid-state drives (SSDs), optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, and ROMs.

[0125] In addition, the program code that realizes the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C / C++, perl, Shell, PHP, Python (registered trademark), and Java (registered trademark).

[0126] Furthermore, the program code of the software that realizes the functions of the embodiment may be distributed via the network 1 and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the processor 201 of the computer may read and execute the program code stored in the storage means or the storage medium.

[0127] In the above embodiment, the control lines and information lines are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. All components may be interconnected.

[0128] The above various explanations can be summarized as follows. The following summary may include supplementary explanations and explanations of variations of the above explanations. In the information processing system 100 of this embodiment, the processor may further execute a process of evaluating the classification based on at least the classification accuracy based on the result of the classification and the calculated amount of calculation.

[0129] This makes it possible to appropriately improve the accuracy of identifying confidential information according to the evaluation results, thereby optimizing the amount of input information in the language model that determines the confidentiality of information, and enabling more efficient and accurate identification of confidential information.

[0130] Furthermore, in the information processing system 100 of this embodiment, the storage device may store information on sentences in the document, such as the sentence body, the number of tokens representing the amount of calculation, and the vector of the embedded expression, and the processor may calculate the amount of calculation based on the information on the number of tokens of the specific sentence and the surrounding sentences, and the vector of the embedded expression.

[0131] This makes it possible to accurately estimate the amount of calculation required for the language model when determining the confidentiality of a sentence, i.e., when identifying confidential information. As a result, it is possible to optimize the amount of input information in the language model that determines the confidentiality of information, thereby enabling more efficient and accurate identification of confidential information.

[0132] In addition, in the information processing system 100 of this embodiment, the storage device may further store information on the threshold value of the certainty, and when generating the surrounding information list, the processor may perform a process of calculating a surrounding information list evaluation value based on the magnitude of the certainty taking into account the threshold value and the low amount of calculation, and a process of optimizing the calculated surrounding information list evaluation value.

[0133] This allows the peripheral information list to be improved in accordance with the evaluation value, thereby optimizing the amount of input information in the language model that determines the confidentiality of information, and enabling more efficient and accurate identification of confidential information.

[0134] Furthermore, in the information processing system 100 of this embodiment, the storage device may further store confidentiality policy information indicating the meaning of sentences that are considered confidential information in a document, and information on vectors of embedded expressions in information on sentences in the document, and when calculating the confidence level, the processor may perform a process of calculating the similarity between the vectors of the embedded expressions in the specific sentence and the surrounding sentences, and obtain the confidence level by adding information corresponding to the input information to a prediction model that uses as input information at least the similarity between the embedded expressions in the confidential information indicated by the confidentiality policy information, the embedded expressions in the specific sentence, and the surrounding sentences, and outputs a predicted value of the confidence level.

[0135] This allows for more appropriate prediction of the confidence level, which in turn allows for the optimization of the amount of input information in the language model that determines the confidentiality of information, thereby enabling more efficient and accurate identification of confidential information.

[0136] Furthermore, in the information processing system 100 of this embodiment, the storage device may further store information on the classification threshold, and when evaluating the classification, the processor may calculate a classification method evaluation value based on at least the classification accuracy based on the classification result and the calculated amount of calculation, and set the confidence threshold based on whether or not the classification method evaluation value has improved between the time the peripheral information list was generated.

[0137] This makes it possible to determine the confidence level using a threshold value that corresponds to various situations, etc. Furthermore, it is possible to optimize the amount of input information in the language model that determines the confidentiality of information, thereby enabling the identification of confidential information with greater efficiency and accuracy. [Explanation of symbols]

[0138] 1 Network 50 Corporate business systems 51 Language Model System 100 Information Processing Systems 101 Surrounding information list generation unit 102 Confidential Information Identification Department 103 Identification Method Evaluation Unit 111 Classification confidence prediction unit 121 Sensitive classification information 122 Policy Information 123 Document Information 124 Sentence Information 125 Confidential Identifying Information 126 Identification Confidence Information 127 Classification method evaluation value information 200 calculator 201 processor 202 Main storage 203 Auxiliary storage device 204 Input Device 205 Display device 206 Communication Equipment 1300 Example input to language model 1400 screen

Claims

1. a storage device that holds information about sentences present in at least a given document; a processor that executes a process of calculating the expected amount of calculation and confidence level when inputting each set of a specific sentence and its surrounding sentences, the confidentiality of which is to be determined, into a language model for confidentiality determination based on information about the sentence, and identifying the confidentiality of the specific sentence; a process of generating a surrounding information list indicating the surrounding sentences selected based on the result of the calculation; and a process of identifying whether the specific sentence is classified information by inputting the specific sentence and the surrounding sentences into the language model based on the surrounding information list. An information processing system comprising:

2. The processor: Further, a process of evaluating the classification is performed based on at least the classification accuracy based on the result of the classification and the calculated amount of calculation.

2. The information processing system according to claim 1, wherein:

3. The storage device includes: The information of the sentence in the document includes information on the sentence body, the number of tokens representing the amount of calculation, and the vector of the embedding expression; The processor: The calculation of the amount of calculation is performed based on the number of tokens in the specific sentence and the surrounding sentences, and information on the vector of the embedded expression.

2. The information processing system according to claim 1, wherein:

4. The storage device includes: Further retaining information about the confidence threshold; The processor: When generating the neighborhood information list, a process of calculating a neighborhood information list evaluation value according to the magnitude of the certainty factor based on the threshold value and the smallness of the amount of calculation, and a process of optimizing the calculated neighborhood information list evaluation value are executed.

2. The information processing system according to claim 1, wherein:

5. The storage device includes: The method further includes retaining confidentiality policy information representing the meaning of sentences that are classified as confidential information in the document and information on vectors of embedded expressions in the sentences in the document; The processor: In calculating the confidence level, a process of calculating a similarity between vectors of the embedded expressions of the specific sentence and the surrounding sentences, and a prediction model that receives as input information at least the similarity between the embedded expressions in the confidential information indicated by the confidentiality policy information, the embedded expressions of the specific sentence, and the surrounding sentences and outputs a predicted value of the confidence level, and obtains the confidence level by adding information corresponding to the input information.

2. The information processing system according to claim 1, wherein:

6. The storage device includes: Further retaining information on the discrimination threshold; The processor: When evaluating the classification, a classification method evaluation value is calculated based on at least the classification accuracy based on the classification result and the calculated amount of calculation, and a threshold value of the degree of certainty is set based on whether or not the classification method evaluation value has improved between the times when the peripheral information list is generated.

3. The information processing system according to claim 2, wherein:

7. The information processing system maintaining, in a storage device, information on sentences present in at least a predetermined document; a process of calculating the amount of calculation and the degree of certainty expected when inputting each set of a specific sentence and its surrounding sentences, the confidentiality of which is to be determined based on information about the sentence, into a language model for confidentiality determination to identify the confidentiality of the specific sentence; a process of generating a list of surrounding information indicating the surrounding sentences selected based on the results of the calculation; and a process of inputting the specific sentence and the surrounding sentences into the language model based on the list of surrounding information to identify whether the specific sentence is confidential information.

2. An information processing method comprising:

8. an information processing device that stores information of sentences present in at least a predetermined document in a storage device; a step of calculating, based on the information of the sentence, for each set of a specific sentence to be judged for confidentiality and its surrounding sentences, the amount of calculation and the degree of certainty expected when inputting the set into a language model for confidentiality judgment to identify the confidentiality of the specific sentence; generating a peripheral information list indicating peripheral sentences selected based on the result of the calculation; a step of inputting the specific sentence and the surrounding sentences into the language model based on the surrounding information list to identify whether the specific sentence is confidential information; A program that executes.

Images