Information processing system, information processing method, and program

Abstract

The present invention optimizes the amount of input information with respect to a language model for determining the confidentiality of information, and enables identification of confidential information with excellent efficiency and accuracy. This information processing system 100 is configured to comprise: an auxiliary storage device 203 for retaining information about sentences; and a processor 201 that calculates, on the basis of the information about the sentences, an anticipated computational cost and certainty factor during confidentiality identification with regard to respective sets of a specific sentence and a peripheral sentence, that generates a peripheral information list indicating peripheral sentences selected on the basis of results thereof, and that inputs specific sentences and peripheral sentences to a language model on the basis of the peripheral information list to thereby identify whether the specific sentences are confidential information.

Description

Information processing system, information processing method, and program

[0001] The present invention generally relates to information processing systems, information processing methods, and program technologies, and more specifically to a technology that optimizes the amount of input information in a language model that determines the confidentiality of information, thereby enabling the identification of confidential information with good efficiency and accuracy.

[0002] For example, confidential information helps a company secure its uniqueness in the market and gain a competitive advantage. Therefore, its owner must take appropriate measures to maintain the confidentiality of the confidential information. In order to properly manage confidential information, information managers in such companies set and manage confidentiality importance classifications, mainly on a document-by-document basis.

[0003] In the above management method, the information manager estimates the extent of damage that would occur if information were leaked based on the type of document and its overall content. Based on the results of this estimation, the information manager then determines the confidentiality importance classification for the document in question according to the degree of confidentiality protection required. However, in reality, there are many cases where confidential and non-confidential information is mixed within each document, making it difficult to accurately identify the confidential information. Furthermore, even if only a small portion of the information in a document is classified, the entire document will be classified.

[0004] In such a case, it becomes difficult to utilize the document, i.e., to utilize non-confidential information in the document. Therefore, a management method is required in which the content referred to in the document is divided into semantic units and confidential information is managed by these semantic units. As a conventional technique for identifying confidential information, for example, a technique has been proposed (see Patent Document 1) for generating a publicly available document image without using confidential information based on a document image containing confidential information.

[0005] This technology relates to an information processing device that includes a means for acquiring character blocks of a first document image obtained by reading a document and character strings obtained by performing character recognition processing on each character block, a means for extracting character strings corresponding to named entities of predetermined attributes from the acquired character strings, and a means for generating document data including at least attribute information related to the attributes of the named entities corresponding to the extracted character strings and position information of the character blocks corresponding to the extracted character strings.

[0006] JP 2023-087840 A

[0007] On the other hand, there are also methods that identify confidential information by preparing registered keywords for confidential information in advance and detecting the registered keywords contained in the target document. However, these methods have the problem of not being able to identify confidential information that contains keywords other than the registered keywords. Furthermore, even in documents in which registered keywords appear, there are cases in which the information is non-confidential depending on the meaning of the sentence, and there is also the problem of insufficient accuracy.

[0008] On the other hand, one method for solving the above problem is to identify confidential information by taking into account the meaning of a document using natural language processing with a large-scale language model (hereinafter simply referred to as a "language model"), as shown in Patent Document 1. When a language model is used, it becomes possible to determine the confidentiality of a sentence to be identified in a document by taking into account the sentence.

[0009] For example, consider a case where a document contains two sentences, such as "Result AA was obtained. This is due to BB.", and the meaning of the sentence regarding "interpretation of the experimental results" needs to be identified as confidential information. In this case, it is difficult to identify the sentence "This is due to BB" on its own as a description regarding "interpretation of the experimental results." Therefore, in order to identify this sentence as confidential information, it is necessary to take into account the meaning of the sentence before (or after) it.

[0010] When using a language model to identify confidential information, in order to take into account the context surrounding the sentence to be identified, it is necessary to input not only the sentence to be identified but also the surrounding information into the language model. Of course, due to the characteristics of language models, if a large amount of surrounding information is input, improvement in identification accuracy can be expected. However, there is a concern that an increase in the amount of input information will increase the amount of calculation. Conversely, if the input of surrounding information is insufficient, there is a concern that identification accuracy will decrease.

[0011] Furthermore, the appropriate amount of peripheral information may vary depending on the sentence to be identified. For example, in the above example, if the sentence to be identified is "The result AA was obtained," it can be easily identified as a description of experimental results without providing any peripheral information, so a small amount of peripheral information is sufficient to input. On the other hand, if the sentence to be identified is "It is BB," it cannot be identified as meaning "an interpretation of the experimental results" without providing any peripheral information, so a certain amount of peripheral information must be input.

[0012] As described above, there is a trade-off between identification accuracy and computational complexity, and the appropriate amount of peripheral information varies depending on various factors, making it difficult to determine. Therefore, the present invention has been made in consideration of the above-mentioned problems, and aims to provide a technology that optimizes the amount of input information in a language model that determines the confidentiality of information, thereby enabling identification of confidential information with good efficiency and accuracy.

[0013] The present application includes multiple means for solving the above-mentioned problems, examples of which are as follows: To solve the above-mentioned problems, an information processing system according to one aspect of the present invention comprises: a storage device that stores information on sentences present in at least a predetermined document; a process that, based on the information on the sentences, calculates an expected amount of calculation and a degree of certainty when inputting each set of a specific sentence to be determined for confidentiality and its surrounding sentences into a language model for confidentiality determination and identifying the confidentiality of the specific sentence; a process that generates a surrounding information list indicating the surrounding sentences selected based on the results of the calculation; and a processor that executes a process of inputting the specific sentence and the surrounding sentences into the language model based on the surrounding information list, thereby identifying whether the specific sentence is confidential information.

[0014] In addition, in order to solve the above problem, an information processing method according to one embodiment of the present invention is characterized in that an information processing system stores information on sentences present in at least a specified document in a storage device, and, based on the information on the sentences, calculates the expected amount of calculation and confidence level when inputting each set of a specific sentence and its surrounding sentences to be judged for confidentiality into a language model for confidentiality judgment to identify the confidentiality of the specific sentence; generates a surrounding information list indicating the surrounding sentences selected based on the results of the calculation; and, based on the surrounding information list, inputs the specific sentence and the surrounding sentences into the language model to identify whether the specific sentence is confidential information.

[0015] In addition, in order to solve the above problem, a program according to one aspect of the present invention is characterized in that it causes an information processing device that stores information on sentences present in at least a specified document in a storage device to execute the following steps: based on the information on the sentences, calculate the expected amount of calculation and confidence level when inputting each set of a specific sentence and its surrounding sentences to be judged for confidentiality into a language model for confidentiality judgment and identifying the confidentiality of the specific sentence; generating a surrounding information list indicating the surrounding sentences selected based on the results of the calculation; and identifying whether the specific sentence is confidential information by inputting the specific sentence and the surrounding sentences into the language model based on the surrounding information list.

[0016] According to the present invention, the amount of input information in a language model for determining the confidentiality of information is optimized, and confidential information can be identified with good efficiency and accuracy.

[0017] 1 is a diagram illustrating an example of a functional configuration of an information processing system according to a first embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing system according to a first embodiment. FIG. 3 is a diagram illustrating an example of confidentiality classification information according to a first embodiment. FIG. 4 is a diagram illustrating an example of policy information according to a first embodiment. FIG. 5 is a diagram illustrating an example of document information according to a first embodiment. FIG. 6 is a diagram illustrating an example of sentence information according to a first embodiment. FIG. 7 is a diagram illustrating an example of confidential identification information according to a first embodiment. FIG. 8 is a diagram illustrating an example of identification certainty information according to a first embodiment. FIG. 9 is a diagram illustrating an example of a flow of an information processing method according to a first embodiment. FIG. 10 is a diagram illustrating an example of a flow of an information processing method according to a first embodiment. FIG. 11 is a diagram illustrating an example of an input to a language model according to a first embodiment. FIG. 12 is a diagram illustrating an example of a functional configuration of an information processing system according to a second embodiment. FIG. 13 is a diagram illustrating an example of identification method evaluation value information according to a second embodiment. FIG. 14 is a diagram illustrating an example of a flow of an information processing method according to a second embodiment. FIG. 15 is a diagram illustrating an example of an output according to a second embodiment.

[0018] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention should not be construed as being limited to the description of the embodiments shown below. Those skilled in the art will readily understand that the specific configuration can be changed without departing from the spirit or intent of the present invention.

[0019] In the configurations of the invention described below, identical or similar components or functions are denoted by the same reference numerals, and redundant explanations are omitted. The designations "first," "second," "third," etc. in this specification are used to identify components and do not necessarily limit the number or order. The position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc., in order to facilitate understanding of the invention. Therefore, the present invention is not limited to the position, size, shape, range, etc. disclosed in the drawings, etc.

[0020] In the following description, a communication device may be one or more communication interface devices, which may be one or more homogeneous communication interface devices (e.g., one or more NICs (Network Interface Cards)) or two or more heterogeneous communication interface devices (e.g., a NIC and an HBA (Host Bus Adapter)).

[0021] In the following description, a "memory" refers to one or more memory devices, which are an example of one or more storage devices. At least one of the memory devices may be a volatile memory device or a non-volatile memory device.

[0022] In the following description, a "storage device" may refer to one or more persistent storage devices, which are an example of one or more storage devices. A persistent storage device may typically be a non-volatile storage device, specifically, for example, a hard disk drive (HDD), a solid state drive (SSD), or a non-volatile memory express (NVMe) drive.

[0023] Furthermore, in the following description, a "CPU" refers to an arithmetic unit and may be one or more processor devices. The at least one processor device may typically be a microprocessor device such as a CPU (Central Processing Unit), but may also be other types of processor devices such as a GPU (Graphics Processing Unit). The at least one processor device may be a single-core or multi-core. The at least one processor device may also be a processor core. The at least one processor device may also be a broader processor device such as a hardware circuit that performs part or all of the processing (for example, an FPGA (Field-Programmable Gate Array), a CPLD (Complex Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit)).

[0024] In the following description, information that provides an output in response to an input may be described using expressions such as "xxx table" or "xxx database." However, this information may be data of any structure (for example, structured data or unstructured data), or may be a learning model such as a neural network, genetic algorithm, or random forest that generates an output in response to an input. Therefore, "xxx table" or "xxx database" can be referred to as "xxx information." In the following description, the structure of each database or table is an example, and one database or table may be divided into two or more databases or tables, or all or part of two or more databases or tables may be a single database or table.

[0025] In the following description, processing may be described using a "program" as the subject. However, since a program is executed by a CPU to perform a predetermined process using a storage device and / or an interface device, etc., as appropriate, the subject of the process may also be the CPU (or a device such as a controller having a processor). A program may be installed in a device such as a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable (e.g., non-transitory) recording medium. In the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0026] In addition, in the following description, when describing elements of the same type without distinguishing between them, common parts of the reference symbols may be used, and when describing elements of the same type with distinction between them, reference symbols or element identifiers may be used.

[0027] <Configuration of Information Processing System> FIG. 1 is a diagram illustrating an example configuration of an information processing system 100 according to a first embodiment. The information processing system 100 according to the first embodiment optimizes the amount of input information in a language model that determines the confidentiality of information, enabling efficient and accurate identification of confidential information. Specifically, the information processing system 100 optimizes peripheral information provided to the language model for each sentence to be identified, taking into account the trade-off between accuracy and computational complexity when using a language model to identify whether a sentence is classified as confidential information. Furthermore, the amount of peripheral information provided can be adjusted appropriately depending on the confidentiality importance, the information acquirer, and the confidentiality policy. According to the present disclosure, when managing confidential information for each sentence, it is possible to identify confidential information with a low computational complexity and high identification accuracy. The following description will be given using confidential information handled in a company as an example of confidential information.

[0028] FIG. 1 is a diagram illustrating an example of a functional configuration of an information processing system 100 according to a first embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer 200 constituting the information processing system 100 according to the first embodiment. The information processing system 100 is composed of computers 200 as shown in FIG. 2. The number of computers 200 constituting the information processing system 100 is not limited. The computer 200 includes a processor 201, a main memory device 202, an auxiliary memory device 203, an input device 204, a display device 205, and a communication device 206. The hardware elements are connected to each other via a bus.

[0029] The input device 204 is a keyboard, mouse, or the like, and accepts input of data and instructions. The information manager operates the input device 204 to instruct the identification of confidential information regarding a specific document. When the information processing system 100 accepts the instruction, it identifies the confidential information contained in the document. The display device 205 is a display, or the like, and displays setting information, processing results, and the like.

[0030] The main storage device 202 is a memory or the like that stores programs executed by the processor 201 and data used by the programs. The main storage device 202 is also used as a work area. The auxiliary storage device 203 is a non-volatile memory device such as an HDD (Hard Disk Drive) that permanently stores programs and information.

[0031] The auxiliary storage device 203 stores programs that realize the peripheral information list generation unit 101 and the confidential information identification unit 102. The auxiliary storage device 203 also has a storage unit 104, which stores confidential classification information 121, policy information 122, document information 123, sentence information 124, confidential identification information 125, and identification certainty information 126.

[0032] The processor 201 operates as a functional unit (module) that realizes a specific function by executing the above program. In the following description, when a process is described using a functional unit as the subject, it indicates that the processor 201 is executing a program that realizes the functional unit.

[0033] The communication device 206 communicates with a business system 50 that manages confidential information via the network 1. The communication device 206 may also be capable of communicating with a language model system 51 via the network 1. In this case, the information processing system 100 calls and uses the language model function of the present invention from the external language model system 51, and obtains the results. Of course, this is just one example, and the implementation and usage of the language model are not limited.

[0034] <Example of Information Configuration> Here, information held by the information processing system 100 will be described with reference to FIGS. 3 to 8 in addition to FIG. 1 . FIG. 3 is a diagram showing an example of the data structure of the confidentiality classification information 121 according to the first embodiment. The confidentiality classification information 121 is information for managing the classification of information treated as confidential. The confidentiality classification information 121 stores entries including a confidentiality classification ID 301, a confidentiality importance classification 302, an information acquirer classification 303, and a policy ID 304. One entry exists for one confidentiality classification. Note that the fields included in an entry are merely examples and are not limited to these. Any of the above-mentioned fields may not be included, or other fields may be included.

[0035] The confidentiality classification ID 301 is a field for storing the ID of the confidentiality classification, and is the same field as the confidentiality classification ID 701 and confidentiality classification ID 801 described later. The confidentiality importance classification 302 is a field for storing information indicating the importance of confidentiality (e.g., public, confidential, secret, top secret, etc.), and is the same field as the confidentiality importance classification 502 described later. The information acquirer classification 303 is a field for storing the classification related to the person acquiring the information (e.g., external customer, internal department, etc.). The policy ID 304 is a field for storing the ID of the confidentiality policy, which is the meaning of what is considered confidential information, and is the same field as the policy ID 401 described later. The confidentiality classification information 121 may be set in advance or acquired from outside.

[0036] 4 is a diagram showing an example of the data structure of the policy information 122 according to the first embodiment. This policy information 122 is information for managing confidentiality policies. The policy information 122 stores entries each including a policy ID 401, a policy content 402, a policy keyword 403, and an embedded expression vector 404. One entry exists for one confidentiality policy. Note that the fields included in an entry are merely examples and are not limiting. Any of the above-mentioned fields may be omitted, or other fields may be included.

[0037] Of these, policy ID 401 is a field for storing the ID of the confidentiality policy. Policy content 402 is a field for storing the content of the confidentiality policy in a free description format. Policy keyword 403 is a field for storing a keyword that matches the content of the confidentiality policy. Embedded expression vector 404 is a field for storing a value obtained by converting policy content 402 into an embedded expression vector. Policy information 122 may be set in advance or obtained from an external source.

[0038] 5 is a diagram showing an example of the data structure of the document information 123 according to the first embodiment. This document information 123 is information used by an information manager to manage documents. The document information 123 stores entries each including a document ID 501, a confidentiality importance classification 502, and a document name 503. One entry exists for one document. Note that the fields included in the entry are merely examples and are not limited to these. Any of the above-mentioned fields may be omitted, or other fields may be included.

[0039] Of these, the document ID 501 is a field that stores the ID of the document, and is the same field as the document ID 602 ​​described below. The confidentiality importance classification 502 is a field that stores information indicating the confidentiality importance. The document name 503 is a field that stores the name of the document. The document information 123 may be set in advance or obtained from an external source.

[0040] FIG. 6 is a diagram showing an example of the data structure of sentence information 124 in Example 1. This sentence information 124 is information for managing sentences in a document. The sentence information 124 stores entries including a sentence ID 601, a document ID 602, text 603, a token count 604, and an embedded expression vector 605. One entry exists for each combination of a sentence ID and a document ID. Note that the fields included in an entry are merely examples and are not limited to these. Any of the above-mentioned fields may be omitted, or other fields may be included.

[0041] Of these, sentence ID 601 is a field that stores the ID of a specific sentence in a document, and is the same field as sentence ID 702 and sentence ID 802, which will be described later. Document ID 602 ​​is a field that stores the ID of the document to which the sentence stored in sentence ID belongs. Text 603 is a field that stores the content of the sentence written in free text format.

[0042] The number of tokens 604 is a field that stores the number of tokens, which are basic units of text obtained by dividing a character string into meaningful chunks. The number of tokens represents the length of the text, and as will be described later, the amount of calculation for the language model is estimated based on this value. The embedded expression vector 605 is a field that stores a value obtained by converting the text 603 written in free description format into an embedded expression vector. The sentence information 124 may be set in advance or obtained from an external source.

[0043] FIG. 7 is a diagram showing an example of the data structure of the confidential identification information 125 according to the first embodiment. This confidential identification information 125 is information for managing an identification result as to whether a certain sentence corresponds to confidential information when considered from the perspective of a specific confidential classification. The confidential identification information 125 stores entries including a confidential classification ID 701, a sentence ID 702, and an identification result 703. One entry exists for each combination of a confidential classification ID and a sentence ID. Note that the fields included in an entry are merely examples and are not limited to these. Any of the above-mentioned fields may be omitted, or other fields may be included.

[0044] Of these, the confidentiality classification ID 701 is a field that stores the ID of the confidentiality classification. The sentence ID 702 is a field that stores the ID of a specific sentence. The identification result 703 is a field that stores whether a certain sentence corresponds to confidential information when considered from the perspective of a specific confidentiality classification. In this embodiment, either "1" indicating that it corresponds to confidential information or "0" indicating that it does not correspond to confidential information is set. The confidentiality identification information 125 may be set in advance or may be obtained from outside.

[0045] 8 is a diagram showing an example of the data structure of the identification certainty information 126 in Example 1. This identification certainty information 126 is information for managing the identification certainty when a sentence is identified as classified information from the perspective of a specific confidentiality classification, and is information for managing lists of peripheral information (peripheral sentences) generated by the information processing system 100. Multiple peripheral information lists may be generated for one combination of sentence and confidentiality classification.

[0046] Of these, the identification certainty information 136 stores entries including a confidentiality classification ID 801, a sentence ID 802, a peripheral information list 803, an identification result 804, an identification certainty 805, and a peripheral information list evaluation value 806. One entry exists for each combination of a confidentiality classification ID, a sentence ID, and a peripheral information list. Note that the fields included in an entry are merely examples and are not limited to these. Any of the above-mentioned fields may not be included, and other fields may also be included.

[0047] Of these, the confidentiality classification ID 701 is a field that stores the ID of the confidentiality classification. The sentence ID 702 is a field that stores the ID of a specific sentence. The peripheral information list 803 is a field that stores, in the form of a list of sentence IDs, sentences of peripheral information that are input to the language model when identifying confidential information. The identification result 804 is a field that stores the identification result of confidential information output by the language model. The identification certainty 805 is a field that stores the identification certainty regarding the identification of confidential information output by the language model. The peripheral information list evaluation value 806 is a field that stores an evaluation value calculated by the information processing system 100 based on the sentence to be identified and the peripheral information list. The identification certainty information 126 may be set in advance or obtained from outside.

[0048] 9 is a flowchart illustrating processing executed by the information processing system 100 according to the first embodiment. When the information processing system 100 according to the first embodiment receives, from the input device 204 or the like, an instruction requesting identification of sentences containing confidential information in a specific document, the information processing system 100 starts the processing described below.

[0049] First, the peripheral information list generation unit 101 of the information processing system 100 receives the sentence to be identified (document information 123 and sentence information 124), information on its peripheral sentences (sentence information 124), and information on the confidentiality policy (policy information 122) in the above-mentioned instructions, etc., and generates a peripheral information list to be input into a language model (for example, used via the language model system 51) used to identify whether the sentence to be identified (specific sentence) contains confidential information. The peripheral information list generation unit 101 includes an identification certainty prediction unit 111.

[0050] Next, the identification certainty prediction unit 111 of the information processing system 100 calculates a prediction value of the identification certainty based on the confidentiality policy, the sentence to be identified, and the peripheral information list. Also, the confidential information identification unit 102 of the information processing system 100 receives information related to the sentence to be identified, the peripheral information list generated by the peripheral information list generation unit 101, and the confidentiality policy, and identifies whether the target sentence contains confidential information.

[0051] Here, an overview of the processing executed by the information processing system 100 of the first embodiment will be described. It is assumed that the document ID of the document to be processed is prepared in advance in the information processing system 100. First, the information processing system 100 determines a confidentiality classification ID for the document to be identified, which is composed of a confidentiality importance classification, an information acquirer classification, and a confidentiality policy (step S100). The confidentiality importance classification is obtained from the document information 123 by the confidentiality importance classification 502 of the document whose document ID matches. The information acquirer classification and confidentiality policy are set by the information administrator depending on the purpose of protecting confidential information. Although multiple information acquirer classifications and confidentiality policies can be selected, in this embodiment, one of each is selected and processing is executed. In this way, the information processing system 100 selects one of the classification IDs listed in the confidentiality classification information 121.

[0052] Next, the information processing system 100 calculates the classification confidence threshold conf ｔｈｒｅｓ (Step S200). ｔｈｒｅｓThe higher the confidence level, the stricter the criteria for identifying confidential information. ｔｈｒｅｓ If the threshold is not higher than , the processing is set so that confidential information is not identified using the language model. Any method for determining the threshold is allowed.

[0053] Next, the information processing system 100 selects a sentence included in the target document (step S300). Such a sentence may be selected from the sentence information 124, any sentence whose document ID 602 ​​matches the document ID of the target document.

[0054] Next, the neighborhood information list generating unit 101 of the information processing system 100 executes a neighborhood information list generating process (step S400). In this neighborhood information list generating process, the identification certainty factor predicting unit 111 executes an identification certainty factor predicting process (step S500).

[0055] Next, the confidential information identification unit 102 of the information processing system 100 executes the confidential information identification process (step S600). The information processing system 100 also determines whether the processes from step S400 to step S600 have been executed for all sentences included in the specified document ID (step S700).

[0056] If the result of the above determination is that the processing has not been performed on all sentences (step S700: No), the information processing system 100 returns to step S300 and similarly performs each step from step S300 onwards. On the other hand, if the result of the above determination is that the processing has been performed on all sentences (step S700: Yes), the information processing system 100 completes the identification processing for the document that is the target of this processing.

[0057] (Neighborhood Information List Generation Process) Next, details of each process executed by the information processing system 100 will be described. Fig. 10 is a flowchart illustrating neighborhood information list generation process executed by the information processing system 100 of the first embodiment. Fig. 11 is a flowchart illustrating identification certainty factor prediction process executed by the information processing system 100 of the first embodiment.

[0058] In this case, the peripheral information list generation unit 101 first acquires (step S401) a confidentiality policy from the confidentiality classification ID selected in processing step S100 executed by the information processing system 100, and the sentence selected in step S300. The confidentiality policy is acquired by acquiring a policy ID 304 from an entry having the same confidentiality classification ID 301 as the confidentiality classification ID selected from the confidentiality classification information 121, and acquiring a policy content 402 and an embedded expression vector 404 from an entry having the same policy ID 401 as the policy ID acquired from the policy information 122.

[0059] Also, for the sake of explanation, if the selected sentence is sentence M, the peripheral information list generation unit 101 obtains information about the sentence having the sentence ID 601 corresponding to sentence M from the sentence information 124, i.e., the number of tokens 604 and the embedded expression vector 605.

[0060] Next, the peripheral information list generation unit 101 acquires information about all sentences that are candidates for the peripheral information list (step S402). In this embodiment, as an example of a candidate for the peripheral information list, the sentences N before and N after the selected sentence are taken as examples. For the sake of explanation, the number of sentences in the entire document is assumed to be L, and M is assumed to be greater than N and greater than L-N.

[0061] Specifically, for all i taking integer values ​​between -N and +N, the peripheral information list generation unit 101 acquires information about sentences having a sentence ID 601 corresponding to sentence M+i from the sentence information 124, i.e., the number of tokens 604 and the embedded expression vector 605. Furthermore, if M is smaller than N, the acquired value is set to "0" for all i for which M+i is smaller than 0. If M is greater than L-N, the acquired value is set to "0" for all i for which M+i is greater than N.

[0062] In the embodiment, for simplicity, the peripheral information for sentence M is set to sentence M+i, but the method for setting the peripheral information is not limited to this. For example, the summary information of the paragraph to which sentence M belongs, the summary information of the paragraphs before and after it, the summary information of the document to which the sentence belongs, and the document name 503 can also be set as peripheral information. In this case, this information can be stored in advance in the sentence information 124 and processed in the same way as the peripheral information sentence M+i.

[0063] Next, the peripheral information list generation unit 101 sets a peripheral information list for the selected sentence (step S403). In this case, the peripheral information list generation unit 101 specifically sets a set of binary variables X_i as information representing the peripheral information list. The binary variable X_i is a variable that is set to "1" if sentence M+i, which is one of the peripheral information for sentence M, is included in the peripheral information list, and is set to "0" if it is not included. The initial peripheral information list may be set randomly or according to any rule. Furthermore, if M is smaller than N, X_i may be set to "0" for all i such that M+i is smaller than 0. If M is greater than L-N, X_i may be set to "0" for all i such that M+i is greater than N. Furthermore, if the peripheral information list evaluation value does not have a pre-update value, it is set to any value.

[0064] Next, the neighborhood information list generation unit 101 calculates an estimated calculation amount based on the number of tokens in the acquired neighborhood information list (step S404). While any method for calculating the estimated calculation amount may be used, in the present disclosure, as an example, the estimated calculation amount is calculated by substituting the estimated calculation amount into Equation 1.

[0065]

[0066] where C is the estimated computational complexity, c １ is a proportionality constant for calculating the amount of calculation from the number of tokens, and T_i represents the number of tokens in sentence M+i.

[0067] The peripheral information list generation unit 101 instructs the identification certainty prediction unit 111 to execute an identification certainty prediction process (step S405). At this time, the peripheral information list generation unit 101 inputs information about the confidentiality policy, the target sentence M, and all sentences from sentences M-N to sentences M+N, which are peripheral information list candidates, to the identification certainty prediction unit 111. Here, the identification certainty prediction process will be described with reference to FIG. 11 .

[0068] The classification certainty prediction unit 111 obtains the embedded expression vectors of the target sentence M and its neighboring information list candidate sentence M+i, and calculates the similarity between sentence M and sentence M+i (step S501). While any method for calculating the similarity may be used, in the present disclosure, as an example, the cosine similarity is calculated by substituting the estimated calculation amount into the following formula 2:

[0069]

[0070] Here, S ｉ is the cosine similarity between sentence M and sentence M+i, B Ｍ is the embedding vector of sentence M, |X| is the magnitude of vector X, and <X·Y> is the dot product of vector X and vector Y.

[0071] The identification certainty factor prediction unit 111 obtains a prediction value of the identification certainty factor using a prediction model prepared in advance (step S502). As an example of the form of the prediction model, in the present disclosure, an embedded expression B of the confidentiality policy is used. Ｐ , the embedding representation B of the target sentence M Ｍ , the inner product value of the target sentence M and its embedding representation S ｉ , and X ｉ is input and a predicted value of the identification certainty is output. Any method for calculating the identification certainty can be used, but in the present disclosure, the identification certainty is calculated by substituting P into the following equation 3, where P is the probability that the output corresponds to confidential information when confidential information is identified using a language model.

[0072]

[0073] Here, conf represents the identification certainty. The closer the probability P of guessing that the information is confidential is to 0 or 1, the larger the value of the above formula 3 becomes. Conversely, if P is close to 0.5 and the probability that the information is confidential is roughly the same as the probability that it is not confidential, the value of formula 3 becomes smaller. Therefore, formula 3 represents the certainty regarding the identification of confidential information.

[0074] The prediction model prepared in advance can be constructed using any model. In this disclosure, construction of a prediction model using a neural network model will be described as an example. The embedded representation B of the confidentiality policy that is the target of this input variable is Ｐ can be obtained from the policy information 122, and the embedded expression B of the sentence M Ｍ , the similarity between the embedded expressions of the target sentence M and the peripheral information sentence M+i, S ｉ , and X ｉ can be obtained from the identification certainty information 126 and the sentence information 124. For one entry of the identification certainty information 126, the embedded representation of the target sentence and its peripheral information list candidate can be obtained from the sentence information 124, and S ｉ It is possible to calculate X ｉ can be obtained from the peripheral information list. To obtain the identification confidence level, which is the output of the model, data related to the identification result can be used as the objective variable. This can be obtained by obtaining the identification result 703 related to the target sentence from the confidential identification information 125. By constructing a learning model using the pairs of input variables and output variables prepared in the above procedure as learning data, it is possible to construct a model that can obtain a predicted value of the identification confidence level.

[0075] In addition, in the embodiment, for simplicity, the input variables of the prediction model are the embedded expression of the confidentiality policy, the embedded expression of the target sentence, the similarity between the embedded expressions of the target sentence and the sentences of the peripheral information, and the peripheral information list, but the way in which the input variables are set is not limited to this. For example, in addition to these, the number of occurrences of important keywords and the number of privacy identifiers included in the target sentence and the peripheral information can be added as input variables.

[0076] The important keyword may be, for example, the policy keyword 403 in the policy information 122. The number of occurrences of the important keywords can be obtained by calculating the number of occurrences of the important keywords contained in the target sentence and the peripheral information based on a keyword list prepared in advance. The privacy information identifier can be obtained by calculating the number of occurrences of the important keywords contained in the target sentence and the peripheral information based on a privacy information identifier detection device prepared in advance.

[0077] In addition, the above-mentioned privacy information identifier detection device can be obtained, for example, using a language model that has been pre-trained to detect privacy information identifiers, separate from the language model described in this disclosure, and this can be used to obtain privacy information identifiers in natural language.

[0078] The identification certainty factor prediction unit 111 outputs the predicted value of the identification certainty factor to the peripheral information list generation unit 101, and ends the identification certainty factor prediction process.

[0079] Returning now to the explanation of Fig. 10, the neighborhood information list generation unit 101 calculates a neighborhood information list evaluation value (step S406). Specifically, the neighborhood information list generation unit 101 calculates the neighborhood information list evaluation value using the identification certainty (the "identification certainty" of the present invention) and the estimated computational amount (the "computational amount" of the present invention). In the present disclosure, an example of this evaluation value is calculated by substituting into the following formula 4.

[0080]

[0081] Here, Eval ｌｉｓｔ is the neighborhood information list evaluation value, A is a sufficiently large constant, conf ｔｈｒｅｓ indicates the threshold value given to the discrimination confidence. U(X) is a step function, which is 1 when X>0 and 0 when X<0. ｌｉｓｔ The smaller the confidence level conf is, the higher the evaluation is. ｔｈｒｅｓ If the second term is higher than , the minimum value is 0, and Eval ｌｉｓｔ The smaller the estimated computational complexity C, the smaller the Eval ｌｉｓｔ becomes smaller.

[0082] The neighborhood information list evaluation value in the first embodiment is an index that takes into consideration the estimated amount of calculation and constraints on the identification certainty.

[0083] In the first embodiment, the peripheral information list evaluation value is determined by substituting it into the above formula 4, but the method of determining the evaluation value is not limited to this. For example, formula 4 is intended to impose a constraint on the magnitude of the identification confidence factor conf, but it is also possible to store the identification confidence factor conf previously calculated by the peripheral information list generation unit 101 as the pre-update value of the identification confidence factor, and to define the evaluation formula so as to impose a constraint on the amount of change between the post-update value and the pre-update value of the identification confidence factor, using the calculated identification confidence factor as the post-update value of the identification confidence factor.

[0084] Next, the neighborhood information list generation unit 101 determines whether the neighborhood information list evaluation value has been sufficiently improved (step S407). Specifically, the updated neighborhood information list evaluation value obtained in step S406 is compared with the pre-update neighborhood information list evaluation value. As an example, if the updated neighborhood information list evaluation value is smaller, the value of the neighborhood information list evaluation value is smaller than a set constant A, and the difference between the updated value and the pre-update value is smaller than a predetermined threshold, it can be determined that the neighborhood information list evaluation value has been sufficiently improved.

[0085] If this condition is met (step S407: Yes), the surrounding area information list generating unit 101 sets the binary variable X ｉ The process ends after obtaining the set of neighboring information as a neighboring information list (step S408). On the other hand, if this condition is not met (step S407: No), the neighboring information list generation unit 101 determines whether the process of generating the neighboring information list in step S403 has been performed a certain number of times or more (step S409).

[0086] If the result of the above determination is that the generation process has not been performed a certain number of times or more (step S409: No), the neighborhood information list generation unit 101 returns to step S403. The neighborhood information list evaluation value is substituted as the pre-update value of the neighborhood information list evaluation value. In this step S403, the neighborhood information list is generated so as to search for the neighborhood information list with the smallest neighborhood information list evaluation value. In the first embodiment, for example, a heuristic optimization method such as a genetic algorithm can be used.

[0087] On the other hand, if the result of the above judgment is that the generation process has been performed a certain number of times or more (step S409: Yes), the surrounding information list generation unit 101 determines that it was difficult to properly define the surrounding information list and terminates the process without obtaining the surrounding information list (step S410).

[0088] As described above, the peripheral information list generation unit 101 can appropriately acquire a peripheral information list by taking into consideration the trade-off between the amount of calculation required to identify whether a target sentence contains confidential information and the certainty of the identification. The identification certainty prediction unit 111 can acquire an identification certainty used when the peripheral information list generation unit 101 calculates a peripheral information list evaluation value.

[0089] 12 is a flowchart illustrating the confidential information identification process executed by the information processing system 100 of Example 1. The peripheral information list and peripheral information list evaluation value output from the peripheral information list generation unit 101 are input to the confidential information identification unit 102. The information processing system 100 also inputs information regarding the confidentiality policy, the target sentence, and the set of sentences in the peripheral information list to the confidential information identification unit 102.

[0090] Here, the confidential information identification unit 102 checks whether a peripheral information list can be acquired (step S601). If the result of the above check is that a peripheral information list cannot be acquired (step S601: No), the confidential information identification unit 102 acquires a manual identification result for the confidential information (step S602). Since the peripheral information list generation unit 101 was unable to generate a peripheral information list that satisfied the conditions regarding calculation amount and identification confidence, and therefore was unable to acquire a peripheral information list, it is difficult to identify whether the target sentence contains confidential information using a language model, so the identification result is acquired manually. However, in order to continue mechanical processing, it is not necessary to require manual input of the identification result in step S602.

[0091] The confidential information identification unit 102 also inputs the confidentiality policy, the target sentence, and the manual identification result as one entry in the identification certainty information 126 (step S603). At this time, the set of sentences in the peripheral information list, the peripheral information list evaluation value, and the identification certainty are all set as empty sets. However, if the manual identification result was not input in step S602, the manual identification result is also input as an empty set.

[0092] On the other hand, if the result of the above check is that the peripheral information list can be obtained (step S601: Yes), the confidential information identification unit 102 inputs text information of the confidentiality policy, the target sentence, and the set of sentences in the peripheral information list into the language model in order to obtain the identification result regarding the confidential information in the language model (step S604). Figure 13 shows an example 1300 of input to the language model.

[0093] The confidential information identification unit 102 also acquires the output of the language model, i.e., the identification result and identification certainty for the confidential information (step S605). The confidential information identification unit 102 inputs the confidentiality policy, the target sentence, the set of sentences in the peripheral information list, the peripheral information list evaluation value, the identification result calculated in step S605, and the identification certainty as one entry in the identification certainty information 126 (step S606).

[0094] As described above, the confidential information identification unit 102 can identify whether the target sentence contains confidential information by providing a confidentiality policy, a target sentence, and a set of sentences in the peripheral information list.

[0095] According to the first embodiment, it is possible to optimize the peripheral information provided to the language model for each sentence to be classified, taking into consideration the trade-off between classification accuracy and calculation amount. However, the application of the first embodiment is not limited to supporting the classification of confidential information. That is, although the first embodiment shows a process for identifying whether a sentence contains specific content that corresponds to confidential information, it can also be used as a process for identifying whether a sentence contains specific content other than confidential information.

[0096] Here, consider a request for proposal (RFP) as an example of a document, and "availability" as an example of specific content. This embodiment also aims to identify whether a sentence in the document mentions "availability" while taking into account the context of the sentence. Furthermore, by calculating the number of sentences mentioning "availability" in the document and determining whether the number exceeds a predetermined threshold, it is possible to determine whether a certain RFP sufficiently mentions "availability."

[0097] Next, the threshold value conf of the classification confidence determined by the information processing system 100 according to the first embodiment ｔｈｒｅｓ The second embodiment will be described focusing on the differences from the first embodiment, such as the addition of a process for determining an appropriate value for the first embodiment.

[0098] 14 is a diagram illustrating an example of the functional configuration of an information processing system 100 according to a second embodiment. The hardware configuration of the computer constituting the information processing system 100 according to the second embodiment is the same as that according to the first embodiment. The second embodiment differs in that the auxiliary storage device 203 stores the distinction method evaluation value information 127. The second embodiment differs in that the auxiliary storage device 203 stores a program that realizes the distinction method evaluation unit 103. The second embodiment differs in the outline of the processing executed by the information processing system 100.

[0099] 15 is a diagram showing an example of the data structure of the identification method evaluation value information 127 according to the first embodiment. The identification method evaluation value information 127 is information for managing evaluation values ​​related to identification methods for confidential information, and is generated by the information processing system 100. The identification method evaluation value information 127 stores entries including a confidentiality classification ID 901, an identification certainty threshold 902, an identification accuracy 903, an estimated calculation amount 904, a manual identification ratio 905, and an identification method evaluation value 906.

[0100] There is one entry for each combination of a confidentiality classification ID and an identification certainty threshold. Note that the fields included in an entry are merely examples and are not limited to these. An entry may not include any of the above-mentioned fields, or may include other fields.

[0101] Among these, the confidentiality classification ID 901 is a field for storing the ID of a confidentiality classification, and is the same field as the confidentiality classification ID 301. The identification certainty threshold 902 is a field for storing the identification certainty threshold, which is an input variable of the information processing system 100. The identification accuracy 903 is a field for storing the identification accuracy calculated by the information processing system 100 based on the confidentiality classification ID and the identification certainty threshold. The estimated calculation amount 904 is a field for storing the estimated calculation amount calculated by the information processing system 100 based on the confidentiality classification ID and the identification certainty threshold. The manual identification ratio 905 is a field for storing the manual identification ratio calculated by the information processing system 100 based on the confidentiality classification ID and the identification certainty threshold. The identification method evaluation value 906 is a field for storing an evaluation value related to the identification method calculated by the information processing system 100 based on the confidentiality classification ID and the identification certainty threshold. The identification method evaluation value information 127 may be set in advance or obtained from an external source.

[0102] 16 is a flowchart illustrating a process executed by the information processing system 100 according to the second embodiment. In the first embodiment, the information processing system 100 starts the process when it receives an instruction to identify sentences containing confidential information in a specific document. In the second embodiment, the information processing system 100 starts the process when it receives an instruction to identify sentences containing confidential information in a specific document. ｔｈｒｅｓWhen an instruction is received to determine an appropriate value according to the confidentiality importance classification, information acquirer classification, manual work ratio, etc., the processing is started.

[0103] Furthermore, in the second embodiment, in addition to the peripheral information list generation unit 101 and the confidential information identification unit 102 used in the first embodiment, a classification method evaluation unit 103 is used. The classification method evaluation unit 103 calculates a classification method evaluation value based on the classification accuracy, estimated calculation amount, and manual classification ratio, which are values ​​calculated by the information processing system 100. By evaluating the classification device using the calculated evaluation value, a classification certainty threshold conf determined by the information processing system 100 is obtained. ｔｈｒｅｓ shall be appropriately determined.

[0104] An outline of the processing executed by the information processing system 100 of the second embodiment will be described. The processing from step S100 to step S600 in the second embodiment is the same as that in the first embodiment. It is determined whether the processing from step S400 to step S600 has been executed for all sentences included in the specified document ID (step S700).

[0105] If the result of the above determination is that the processing has not been performed on all sentences (step S700: No), the distinction method evaluation unit 103 returns to step S300 and similarly performs the processing from step S300 onwards. On the other hand, if the result of the above determination is that the processing has been performed on all sentences (step S700: Yes), the distinction method evaluation unit 103 performs a distinction method evaluation process (step S800).

[0106] The discrimination method evaluation unit 103 then determines whether the discrimination method evaluation value has been sufficiently improved (step S900). If the result of the determination is that the discrimination method evaluation value has not been sufficiently improved (step S900: No), the discrimination method evaluation unit 103 returns to step S200 and executes the subsequent processes in the same manner. In step S200, the discrimination method evaluation unit 103 determines the discrimination confidence threshold conf at which the discrimination method evaluation value is minimized. ｔｈｒｅｓ The threshold conf of the classification confidence is set to search for ｔｈｒｅｓ In the second embodiment, for example, an optimization method such as Bayesian optimization is used to generate a threshold conf ｔｈｒｅｓ can be explored.

[0107] If the result of the above determination is that the classification method evaluation value has been sufficiently improved (step S900: Yes), the classification method evaluation unit 103 completes the process and sets the classification confidence threshold conf ｔｈｒｅｓ Determine.

[0108] 17 is a flowchart illustrating an identification method evaluation process executed by the information processing system 100 according to the second embodiment. In this flow, when the information processing system 100 executes the confidential information identification process, all entries stored in the identification certainty information 126 are input to the identification method evaluation unit 103.

[0109] In this case, the identification method evaluation unit 103 calculates the identification accuracy (step S801). To do this, the identification result 804 is obtained for each entry of the identification certainty information 126 input to the identification method evaluation unit 103, and an entry of the classified identification information 125 having the same sentence ID and confidentiality classification ID is searched for to obtain its identification result 703. In the second embodiment, as an example of the identification accuracy, the accuracy rate, which is the rate at which both identification results match, is calculated by substituting the results into the following formula 5:

[0110]

[0111] Here, Accuracy represents the accuracy rate, and L represents the total number of entries of the identification certainty information 126 input to the identification method evaluation unit 103. True_j represents "1" if, for the j-th entry of the identification certainty information 126 input to the identification method evaluation unit 103, the identification result of the identification certainty information 126 matches the identification result of the confidential identification information 125 having the same sentence ID and confidentiality classification ID, and represents "0" if they do not match.

[0112] The discrimination method evaluation unit 103 also calculates an estimated computational amount (step S802). To this end, for each entry of the discrimination certainty information 126 input to the discrimination method evaluation unit 103, the sentence ID of each entry and the number of tokens for the sentence ID in the peripheral information list are obtained from the sentence information 124. Any method can be used to calculate the estimated computational amount based on the number of tokens, but in the present disclosure, as an example, the estimated computational amount is calculated by substituting it into the following formula 6.

[0113]

[0114] where Calc is the estimated computational complexity, c １ is a proportional constant for calculating the amount of calculation from the number of tokens, T ｊ＋ｉ represents the number of tokens in sentence j+i. ｉｊ is a binary variable that indicates whether or not the jth entry sentence of the classification certainty information 126 input to the classification method evaluation unit 103 contains peripheral information i. For the sake of explanation, the number of sentences in the entire document is assumed to be L, so if j+i is smaller than 0 or if j+i is greater than L, the number of tokens is set to "0".

[0115] Next, the classification method evaluation unit 103 calculates the manual classification ratio (step S803). In the second embodiment, the manual classification ratio can be calculated by counting the ratio of the neighbor information list being an empty set for each entry of the classification certainty information 126 input to the classification method evaluation unit 103. In the present disclosure, the manual classification ratio can be calculated by substituting into the following Equation 7.

[0116]

[0117] Here, Work_rate represents the manual identification rate, and Empty_j represents "1" if the peripheral information list of the jth entry of the identification certainty information 126 input to the identification method evaluation unit 103 is an empty set, and "0" if it is not an empty set.

[0118] The classification method evaluation unit 103 calculates a classification method evaluation value taking into consideration the classification accuracy rate, the estimated calculation amount, and the manual classification ratio (step S804). In the present disclosure, as an example, the classification method evaluation value is calculated by substituting it into the following formula 8.

[0119]

[0120] Here, Eval ｍｅｔｈｏｄ represents the classification method evaluation value. α, β, and γ are proportional constants assigned to the accuracy rate Accuracy, the estimated calculation amount Calc, and the manual classification rate Work_rate, respectively. ｍｅｔｈｏｄ is the classification confidence threshold conf ｔｈｒｅｓThe threshold value of the classification confidence is conf ｔｈｒｅｓ When the above is set, the accuracy rate Accuracy, the estimated calculation amount Calc, and the manual classification rate Work_rate are determined, and these are used as the classification confidence threshold conf ｔｈｒｅｓ This is because it changes depending on ｍｅｔｈｏｄ means that the smaller the value, the higher the evaluation. The evaluation value becomes smaller when the accuracy rate Accuracy becomes higher, the estimated calculation amount Calc becomes smaller, or the manual classification rate Work_rate becomes smaller. The classification method evaluation value in Example 2 is an index that takes into account the classification accuracy, the estimated calculation amount, and the manual classification rate.

[0121] 18 is a diagram showing an example of a screen presented by the information processing system 100 of the second embodiment. The screen 1400 of the second embodiment is also displayed in the same manner as in the first embodiment. However, unlike the screen in the second embodiment, the screen displayed in the first embodiment does not include an evaluation value output field 1402. Specifically, the screen 1400 in the second embodiment displays a peripheral information list evaluation value, a peripheral information list, a classification method evaluation value, and a classification confidence threshold.

[0122] The information processing systems 100 of Examples 1 and 2 can be used in a variety of ways. For example, by using the information processing system 100, it is possible to pre-screen whether a document created within an organization contains confidential information before the document is made public or before an email is sent to an information acquirer with a different management classification, and to present any areas that need to be corrected. Furthermore, by using the information processing system 100, it is possible to identify non-confidential information that has previously been treated as confidential information. By appropriately managing and utilizing this information according to the information acquirer, it is possible to improve the accuracy of interactive AI while protecting confidentiality, for example, and more efficiently support investigation work.

[0123] The present invention is not limited to the above-described embodiments, but includes various modifications. For example, the above-described embodiments are provided to explain the present invention in detail, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, some of the configurations of each embodiment can be added to, deleted from, or replaced with other configurations.

[0124] Furthermore, some or all of the above-described configurations, functions, processing units, processing means, etc. may be implemented in hardware, for example, by designing them as integrated circuits. The present invention can also be realized by software program code that implements the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a computer, and the processor 201 of the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium implements the functions of the above-described embodiments, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media for providing such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, solid-state drives (SSDs), optical disks, magneto-optical disks, CD-Rs, magnetic tape, non-volatile memory cards, and ROMs.

[0125] In addition, the program code that realizes the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C / C++, perl, Shell, PHP, Python (registered trademark), and Java (registered trademark).

[0126] Furthermore, the program code of the software that realizes the functions of the embodiment may be distributed via the network 1 and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the processor 201 of the computer may read and execute the program code stored in the storage means or the storage medium.

[0127] In the above embodiment, the control lines and information lines are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. All components may be interconnected.

[0128] The above various explanations can be summarized as follows. The following summary may include supplementary explanations and explanations of variations of the above explanations. In the information processing system 100 of this embodiment, the processor may further execute a process of evaluating the classification based on at least the classification accuracy based on the result of the classification and the calculated amount of calculation.

[0129] This makes it possible to appropriately improve the accuracy of identifying confidential information according to the evaluation results, thereby optimizing the amount of input information in the language model that determines the confidentiality of information, and enabling more efficient and accurate identification of confidential information.

[0130] Furthermore, in the information processing system 100 of this embodiment, the storage device may store information on sentences in the document, such as the sentence body, the number of tokens representing the amount of calculation, and the vector of the embedded expression, and the processor may calculate the amount of calculation based on the information on the number of tokens of the specific sentence and the surrounding sentences, and the vector of the embedded expression.

[0131] This makes it possible to accurately estimate the amount of calculation required for the language model when determining the confidentiality of a sentence, i.e., when identifying confidential information. As a result, it is possible to optimize the amount of input information in the language model that determines the confidentiality of information, thereby enabling more efficient and accurate identification of confidential information.

[0132] In addition, in the information processing system 100 of this embodiment, the storage device may further store information on the threshold value of the certainty, and when generating the surrounding information list, the processor may perform a process of calculating a surrounding information list evaluation value based on the magnitude of the certainty taking into account the threshold value and the low amount of calculation, and a process of optimizing the calculated surrounding information list evaluation value.

[0133] This allows the peripheral information list to be improved in accordance with the evaluation value, thereby optimizing the amount of input information in the language model that determines the confidentiality of information, and enabling more efficient and accurate identification of confidential information.

[0134] Furthermore, in the information processing system 100 of this embodiment, the storage device may further store confidentiality policy information indicating the meaning of sentences that are considered confidential information in a document, and information on vectors of embedded expressions in information on sentences in the document, and when calculating the certainty, the processor may perform a process of calculating the similarity between the vectors of the embedded expressions in the specific sentence and the surrounding sentences, and obtain the certainty by adding information corresponding to the input information to a prediction model that uses as input information at least the similarity between the embedded expressions in the confidential information indicated by the confidentiality policy information, the embedded expressions in the specific sentence, and the surrounding sentences, and outputs a predicted value of the certainty.

[0135] This allows for more appropriate prediction of the confidence level, which in turn allows for the optimization of the amount of input information in the language model that determines the confidentiality of information, thereby enabling more efficient and accurate identification of confidential information.

[0136] In addition, in the information processing system 100 of this embodiment, the storage device may further store information on the classification threshold, and when evaluating the classification, the processor may calculate a classification method evaluation value based on at least the classification accuracy based on the classification result and the calculated amount of calculation, and set the confidence threshold based on whether or not the classification method evaluation value has improved between the time the peripheral information list was generated.

[0137] This makes it possible to determine the confidence level using a threshold value that corresponds to various situations, etc. Furthermore, it is possible to optimize the amount of input information in the language model that determines the confidentiality of information, thereby enabling the identification of confidential information with greater efficiency and accuracy.

[0138] 1 Network 50 Corporate business system 51 Language model system 100 Information processing system 101 Peripheral information list generation unit 102 Confidential information identification unit 103 Identification method evaluation unit 111 Identification certainty prediction unit 121 Confidential classification information 122 Policy information 123 Document information 124 Sentence information 125 Confidential identification information 126 Identification certainty information 127 Identification method evaluation value information 200 Computer 201 Processor 202 Main memory device 203 Auxiliary memory device 204 Input device 205 Display device 206 Communication device 1300 Example of input to language model 1400 Screen

Claims

1. An information processing system comprising: a storage device that stores information on at least sentences present in a specified document; a process that, based on the information on the sentences, calculates the expected amount of calculation and confidence level when inputting each set of a specific sentence and its surrounding sentences to be judged for confidentiality into a language model for confidentiality judgment to identify the confidentiality of the specific sentence; a process that generates a surrounding information list indicating surrounding sentences selected based on the results of the calculation; and a processor that executes a process to identify whether the specific sentence is confidential information by inputting the specific sentence and the surrounding sentences into the language model based on the surrounding information list.

2. The information processing system according to claim 1, characterized in that the processor further executes a process of evaluating the classification based on at least the classification accuracy based on the result of the classification and the calculated amount of calculation.

3. The information processing system of claim 1, wherein the storage device holds information on sentences in the document, such as the sentence body, the number of tokens representing the amount of calculation, and the vector of embedded expressions, and the processor, when calculating the amount of calculation, calculates the amount of calculation based on the information on the number of tokens of the specific sentence and the surrounding sentences, and the vector of embedded expressions.

4. The information processing system described in claim 1, characterized in that the storage device further stores information on the threshold value of the certainty, and the processor, when generating the surrounding information list, performs a process of calculating a surrounding information list evaluation value based on the magnitude of the certainty taking into account the threshold value and the low amount of calculation, and a process of optimizing the calculated surrounding information list evaluation value.

5. The information processing system of claim 1, wherein the storage device further stores confidentiality policy information indicating the meaning of sentences considered confidential information in the document and information on vectors of embedded expressions in information on sentences in the document, and the processor, when calculating the confidence level, performs a process of calculating the similarity between the vectors of the embedded expressions in the specific sentence and the surrounding sentences, and obtains the confidence level by adding information corresponding to the input information to a prediction model that uses as input information at least the similarity between the embedded expressions in the confidential information indicated by the confidentiality policy information, the embedded expressions in the specific sentence, and the surrounding sentences and outputs a predicted value of the confidence level.

6. The information processing system described in claim 2, characterized in that the storage device further stores information on the discrimination threshold, and the processor, when evaluating the discrimination, calculates a discrimination method evaluation value based on at least the discrimination accuracy based on the discrimination result and the calculated amount of calculation, and sets the confidence threshold based on whether the discrimination method evaluation value has improved between the time the peripheral information list was generated.

7. An information processing method characterized by the information processing system storing in a storage device information on sentences present in at least a specified document, and performing the following processes: a process of calculating, based on the information on the sentences, the expected amount of calculation and degree of certainty when inputting each set of a specific sentence to be judged for confidentiality and its surrounding sentences into a language model for confidentiality judgment to identify the confidentiality of the specific sentence; a process of generating a surrounding information list indicating surrounding sentences selected based on the results of the calculation; and a process of identifying whether the specific sentence is confidential information by inputting the specific sentence and the surrounding sentences into the language model based on the surrounding information list.

8. A program that causes an information processing device that stores information on sentences present in at least a specified document in a storage device to execute the following steps: based on the information on the sentences, calculate the expected amount of calculation and confidence level when inputting each set of a specific sentence to be judged for confidentiality and its surrounding sentences into a language model for confidentiality judgment to identify the confidentiality of the specific sentence; generating a surrounding information list indicating the surrounding sentences selected based on the results of the calculation; and identifying whether the specific sentence is confidential information by inputting the specific sentence and the surrounding sentences into the language model based on the surrounding information list.

Images