Information processing system, control system, machine learning method, learned model and information processing method

Abstract

To provide an "information processing system, a control system, a machine learning method, a learned model and an information processing method" capable of making effective preparation about an interview to be performed in assessment.SOLUTION: A server information processing unit 10 of an information processing server 1 is provided with a function for reflecting context information including assessor use information as context to construct a state in which a question can be generated, the assessor use information used when an assessor performed assessment according to a prescribed assessment standard in one past project by a language model GM which is to be used to generate a question to be performed by the assessor according to the prescribed assessment standard about a project of system development.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing system, a control system, a machine learning method, a trained model, and an information processing method, and is particularly suitable for use in an information processing system, a control system, a machine learning method, and an information processing method for assessing a system development project. [Background technology]

[0002] Conventionally, assessors assess system development projects in accordance with a predetermined assessment standard. Examples of predetermined assessment standards include "Automotive SPICE (A-SPICE)" (registered trademark) or CMMI (registered trademark). Typically, in this type of assessment, the assessor interviews the developer during the assessment process. During the interview, the assessor presents various questions related to the assessment to the developer and obtains responses from the developer. The assessor then uses the responses from the developer for various evaluations in accordance with the predetermined assessment standard. This type of assessment quantitatively evaluates the project in accordance with the framework of the predetermined assessment standard, enabling the system development client to evaluate the project objectively and with high accuracy and making it easier for the developer to improve the quality of the project.

[0003] Regarding this type of assessment, Patent Document 1 describes the following technology. Specifically, the assessment support device groups similar questions and confirmation items for each question and confirmation item in a "standard data knowledge database 1, which stores predetermined questions and confirmation items for conducting an assessment." The assessment support device sets and manages the importance of each question and confirmation item in the group in an assessment data knowledge database 2 for each assessment subject to be conducted. The assessment support device outputs the grouped questions and confirmation items in a table format. The above technology is described. Patent Document 1 enables assessments to be conducted efficiently and in a short period of time. Patent Document 2 also describes the following technology. Specifically, an information processing device provides a user with personalized questions as audio and obtains recorded answer audio. Personalized questions are questions for which there is no correct answer that is common to all users. The information processing device generates new answerable questions based on the audio answers. The above technology is described. Patent Document 3 also describes the following technology. That is, the writing creation device 100 has a function of outputting a question sentence to the user relating to the slot to which the value belongs in order to extract multiple values ​​from the answer sentence by the user.The writing creation device 100 then refers to the user's proficiency (ability to have a dialogue necessary for providing a service with a chatbot) and determines the number of slots to include in one question sentence.The above technology is described. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2004-171293 [Patent Document 2] Japanese Patent Application Publication No. 2019-020775 [Patent Document 3] Patent Publication No. 2021-149141 Summary of the Invention [Problem to be solved by the invention]

[0005] As mentioned above, assessments involve interviews by assessors, and the answers provided by the assessee in these interviews are directly related to the assessor's evaluation of the project. Therefore, there was a need for effective advance preparation so that the assessor could respond appropriately to the questions.

[0006] The present invention has been made to solve such problems, and aims to enable effective preparation for interviews conducted in assessments. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, the present invention creates a state in which a language model can generate questions by reflecting, as context, context information including assessor usage information used by an assessor when assessing in accordance with a specified assessment standard in a past project. [Effects of the Invention]

[0008] The present invention configured as described above provides the following advantages. Specifically, a language model is used to generate questions to be asked by an assessor in an assessment. If the questions generated by the language model are inaccurate and off-topic, preparing for an interview using the generated questions cannot be considered effective. On the other hand, the more accurate the questions generated, the more effective the interview preparation using the generated questions. Note that question accuracy refers to the degree of consistency with the questions actually posed by the assessor in the interview; the higher the degree of consistency, the higher the accuracy. According to the present invention, a state is established in which the language model can generate questions by reflecting assessor usage information as context. It is assumed that this assessor usage information includes "information about the project and information about the assessment" necessary for the appropriate execution of the assessment. Therefore, the questions generated by the language model reflect the substance of the project and the substance of the assessment, and are highly consistent with the questions actually posed by the assessor in the interview, and therefore are expected to be highly accurate. Based on the above, the present invention enables effective preparation for interviews conducted in assessments. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of the configuration of an assessment-related system according to a first embodiment. [Figure 2] A block diagram showing an example of the functional configuration of the information processing server and the assessee terminal in the first embodiment. [Figure 3] FIG. 10 is a diagram illustrating an example of the contents of basic learning data. [Figure 4] FIG. 10 is a diagram showing an example of the contents of assessment-compatible learning data. [Figure 5] FIG. 10 is a diagram illustrating an example of the contents of a question learning dataset. [Figure 6] FIG. 2 is a diagram illustrating an example of the operation of the assessment-related system according to the first embodiment. [Figure 7] FIG. 1 illustrates an example of a chat room. [Figure 8]FIG. 1 illustrates an example of a chat room. [Figure 9] FIG. 1 illustrates an example of a chat room. [Figure 10] A block diagram showing an example of the functional configuration of the information processing server and the assessee terminal in the second embodiment. [Figure 11] FIG. 10 is a diagram illustrating an example of the contents of a comprehensive question learning dataset. [Figure 12] FIG. 10 is a diagram illustrating an example of the operation of the assessment-related system according to the second embodiment. [Figure 13] FIG. 10 is a diagram illustrating a modified example of the configuration of an assessment-related system. [Figure 14] FIG. 10 is a diagram illustrating an example of the operation of an assessment-related system according to a modified example. [Figure 15] FIG. 1 illustrates an example of the configuration of an information processing system. DETAILED DESCRIPTION OF THE INVENTION

[0010] First Embodiment A first embodiment of the present invention will be described below with reference to the drawings. Fig. 1 is a diagram showing an example of the configuration of an assessment-related system 1 (information processing system, control system) according to this embodiment. As shown in Fig. 1, the assessment-related system 1 includes an information processing server 2 (information processing system, control system, computer) and an assessee terminal 3 (terminal, computer). Both the information processing server 2 and the assessee terminal 3 can be connected to a network N including the Internet, a telephone network, and other communication networks.

[0011] The information processing server 2 is a server device connected to the network N. The information processing server 2 has a function of executing processing related to assessments pertaining to A-SPICE. A-SPICE will be described later. In FIG. 1 and FIG. 2 described later, the information processing server 2 is represented by a single block. However, this does not mean that the information processing server 2 is composed of a single server device. For example, the information processing server 2 may be composed of multiple server devices. In this case, the server devices constituting the information processing server 2 may include a web server or a web application server. Also in this case, the information processing server 2 may be composed of multiple server devices whose load is distributed by a load balancer. In this case, each of the "functional units that execute information processing" of the multiple server devices functions as an "information processing unit."

[0012] The assessee terminal 3 is a terminal used by the assessee. The assessee will be described later together with an explanation of A-SPICE. The assessee terminal 3 may be of any type. For example, a tablet computer (including so-called smartphones), a desktop computer, or a notebook computer can function as the assessee terminal 3.

[0013] Fig. 2 is a block diagram showing an example of the functional configuration of the information processing server 2 and the assessee terminal 3. As shown in Fig. 2, the information processing server 2 has, as its functional configuration, a server information processing unit 10 (information processing unit), a server communication unit 11, and a server storage unit 12. The assessee terminal 3 has, as its functional configuration, a terminal information processing unit 13 (information processing unit), a terminal communication unit 14, a terminal display unit 15, a terminal input unit 16, and a terminal storage unit 17.

[0014] The server information processing unit 10 includes a processing device including a processor and a primary storage device including RAM. The server information processing unit 10 executes processing (information processing) by having the processing device load a program stored in the storage area of ​​the server storage unit 12 (or another storage area) into the primary storage device and execute it. In other words, the server information processing unit 10 executes processing through cooperation between hardware and software. The server communication unit 11 includes a communication device including a communication control device and a network interface. The server communication unit 11 communicates with external devices via the communication device under the control of the server information processing unit 10. Hereinafter, it is assumed that communication by the information processing server 2 is appropriately executed by the server communication unit 11, and a description of the communication will be omitted. The server storage unit 12 stores data in non-volatile memory. The non-volatile memory is, for example, a hard disk drive (or other magnetic storage device), ROM, or flash memory.

[0015] As shown in Fig. 2, the present language model GM is stored in the server storage unit 12. The present language model GM is a large-scale language model (LLM). As will be apparent later, the server information processing unit 10 has a function of performing machine learning (fine-tuning) on ​​the present language model GM and a function of providing a pseudo-assessment service using the present language model GM.

[0016] The terminal information processing unit 13 comprises a processing device including a processor and a primary storage device including RAM. The terminal information processing unit 13 executes processing (information processing) by having the processing device read out a program stored in the storage area of ​​the terminal storage unit 17 (or another storage area) into the primary storage device and execute it. In other words, the terminal information processing unit 13 executes processing through cooperation between hardware and software. The terminal communication unit 14 comprises a communication device including a communication control device and a network interface. The terminal communication unit 14 communicates with external devices via the communication device under the control of the terminal information processing unit 13. Hereinafter, communication by the assessee terminal 3 is assumed to be appropriately executed by the terminal communication unit 14, and a description of communication will be omitted. The terminal display unit 15 comprises a liquid crystal panel, an organic EL panel, or other display device. The terminal display unit 15 displays an image on the display device under the control of the terminal information processing unit 13. The terminal input unit 16 detects input to the input device and outputs the detection result to the terminal information processing unit 13. The input device is, for example, a keyboard, a mouse, a touch panel, or a camera. The terminal storage unit 17 The device includes a nonvolatile memory and stores data in the nonvolatile memory.

[0017] As shown in Figure 2, a dedicated application AP (hereinafter referred to as the "dedicated application AP") is installed on the assessee terminal 3. The dedicated application AP is dedicated software that implements functions to provide various screens, send and receive various information to and from the information processing server 2, and perform other processes. While the dedicated application AP is running, the terminal information processing unit 13 basically executes processing using the functions of the dedicated application AP (including the terminal's OS, web applications that the dedicated application AP can use, and other programs that can cooperate with the dedicated application AP). The terminal information processing unit 13 communicates with the information processing server 2 and other external devices as needed to obtain information required for processing. The terminal information processing unit 13 also communicates with the information processing server 2 and other external devices as needed to request the execution of processing and obtain the processing results.

[0018] ●Explanation of A-SPICE Next, A-SPICE will be described. In the following description of A-SPICE, the parts of A-SPICE related to this embodiment, particularly the terms used in this embodiment regarding A-SPICE, will be described.

[0019] As is well known, A-SPICE is an assessment standard for in-vehicle software development (system development). In this embodiment, a series of in-vehicle software development projects that are subject to assessment in accordance with A-SPICE is called a "project." In the following, when simply referring to an assessment, it means an assessment that complies with A-SPICE unless otherwise specified.

[0020] In this embodiment, an assessor refers to the entity that conducts the assessment. The assessor is a concept that includes the organization in charge of the assessment and individuals belonging to that organization (especially those who conduct the interviews described below). In addition, in this embodiment, the assessee refers to the entity that carries out the project that is the subject of the assessment. In other words, the assessee refers to the entity that receives the assessment by the assessor. The assessee is a concept that includes the organization that carries out the project and individuals belonging to that organization (especially those who receive the interviews described below).

[0021] In A-SPICE, a project is considered to consist of multiple processes (32 in version 3.1 and version 4) defined in the process reference model. Each process is assessed for its process maturity (sometimes called capability level or process level) in the range of levels L0 to L5. One of the purposes of the assessment is to assess the process maturity of each of the processes that make up the project. Below, individual processes may be represented by their A-SPICE identification information (ID). For example, "SWE.2: Software architecture design" may be simply represented as SWE.2. The same applies to process attributes, base practices, and generic practices, where terms may be represented by their identification information.

[0022] Process attributes (PA) are defined for processes. Process attributes are a set of criteria for judging the capability of a process. The process maturity described above is assessed based on the degree to which the process attributes are achieved. Specific process attributes are as follows: PA1.1: Process Implementation PA2.1: Implementation Management PA2.2: Deliverable management PA3.1: Process Definition PA3.2: Process Deployment PA4.1: Quantitative analysis PA4.2: Quantitative Control PA5.1: Process Innovation PA5.2: Process Innovation Implementation Process maturity: Level L1 corresponds to PA1.1, and level L1 assessment is possible by achieving a certain level of PA1.1. Process maturity: Level L2 corresponds to PA2.1 and PA2.2, and level L2 assessment is possible by achieving a certain level of PA2.1 and PA2.2, assuming a certain level of PA1.1 is achieved. Similarly, level L3 corresponds to PA3.1 and 3.2, level L4 corresponds to PA4.1 and 4.2, and level L5 corresponds to PA5.1 and 5.2.

[0023] Base practices (BP) are defined for processes. Base practices are applied at Process Maturity Level L1 (PA1.1). Base practices define the "tasks, activities, indicators or practices" required to achieve the purpose of the corresponding process. Hereafter, "tasks, activities, indicators or practices" will be simply referred to as "tasks." A rating value (a value indicating the degree of task achievement) for a base practice is derived based on the achievement level of the task. In other words, each base practice is an object (unit) for which a rating value is derived. For example, the following base practices are defined for one of the processes, "SWE.2: Software architectural design." SWE.2.BP1: Create software architecture design document SWE.2.BP2: Allocation of software requirements SWE.2.BP3: Define interfaces of software elements SWE.2.BP4: Description of dynamic behavior SWE.2.BP5: Define resource consumption targets SWE.2.BP6: Evaluate software architecture options SWE.2.BP7: Establish bidirectional traceability SWE.2.BP8: Ensure consistency SWE.2.BP9: Communicate agreed upon software architectural design

[0024] Generic practices (GP) are also defined for processes. Generic practices are applied at process maturity level L2 and above. Generic practices define the tasks required to achieve the purpose of the corresponding process attribute. A rating value for a generic practice is derived based on the degree to which the task is achieved. In other words, each generic practice is an object (unit) from which a rating value is derived. For example, the following generic practices are defined for process attribute PA2.1: GP2.1.1: Identify objectives for the performance of the process GP2.1.2: Plan the performance of the process to accomplish the identified objectives GP2.1.3: Monitor the performance of the process according to the plan GP2.1.4: Coordinate process performance GP2.1.5: Define responsibilities and authorities for performing the process GP2.1.6: Identify, prepare, and make available resources to perform the process according to plan. GP2.1.7: Manage liaison between stakeholders

[0025] Here, the base practices and generic practices each function as the smallest unit from which a rating value is derived. Below, each generic practice and each base practice will be collectively referred to as the "assessment unit." For example, "SWE.2.BP1 (Process: SWE.2 Base Practice: BP1)" corresponds to one assessment unit, and similarly, "SWE.2.GP2.1.1 (Process: SWE.2 Generic Practice GP2.1.1)" corresponds to one assessment unit. The identification information for an assessment unit ("SWE.2.GP2.1.1" in the above example) is called the assessment unit ID.

[0026] During an assessment, the assessor interviews the assessee as appropriate. The interview basically involves the assessor providing questions to the assessee, who then answers the questions. In particular, the assessor conducts an interview for each assessment unit (i.e., for each base practice / generic practice) to obtain the information necessary to derive a rating for that assessment unit. Hereinafter, an interview conducted by the assessor for one assessment unit will be referred to as a "unit interview." For example, the assessor conducts a unit interview related to SWE.2.BP1 to obtain the information necessary to derive a rating for SWE.2.BP1. The assessor uses the information obtained in the unit interview to determine the degree of achievement of the SWE.2.BP1 task and derives a rating.

[0027] A unit interview has the following characteristics. Hereinafter, the characteristics of a unit interview are referred to as "interview characteristics."

[0028] · Interview characteristic F1: In unit interviews, assessors set the content of questions to reflect the characteristics of the project (hereinafter referred to as "project characteristics"). Project characteristics include special circumstances related to the project, special background of the project, points to note about the project, and other project characteristics / attributes. In particular, in this embodiment, the project characteristics of a project can be derived based on the "Assessment Plan 27" (described later) for the project. Project characteristics include, for example, "normal," "a specific grade of functional safety" (e.g., ASIL-A or ASIL-D), "the length of the development period" (long / short), "whether or not it meets the criteria for Proof of Concept (PoC)," or a combination of these. An example of such a combination is "ASIL-D and a short development period." For example, suppose a project characteristic is "short development period." In this case, the assessor sets questions in the unit interviews taking into account the short development period, rather than ignoring the short development period.

[0029] · Interview characteristic F2: Regarding the unit interview for one unit evaluation, the assessor basically asks questions in accordance with the confirmation themes for each of the confirmation themes of that one evaluation unit. For each evaluation unit, confirmation themes are set in the unit interview. Confirmation themes refer to the themes that should be confirmed in the unit interview, and are determined taking into consideration the requirements of A-SPICE and the confirmation items necessary to derive a rating value. For example, for SWE.2.BP1, the confirmation themes are "setting goals," "process planning," "process monitoring," "process adjustment," "defining responsibilities and authorities," "identifying and preparing resources," and "managing interfaces between stakeholders." Similarly, for MAN.3.BP1, the confirmation themes are "project goals," "motivation," and "boundary identification." In a unit interview for one evaluation unit, the assessor basically asks questions in accordance with the confirmation themes for that evaluation unit and obtains answers to the questions.

[0030] ·Interview characteristic F3: The assessor will ask the person being assessed to submit work products as necessary, check the content, and then ask further questions based on the content of the work products. A deliverable is an outcome created during the system development process in a project. A deliverable typically consists of documents. However, deliverables are not limited to documents. Examples of deliverables include a conceptual design document, a detailed design document, an architecture design document, or an interface definition document. At the request of A-SPICE or for the purpose of obtaining information necessary for an appropriate evaluation of the evaluation unit, the assessor will require the submission of deliverables as necessary, check their contents, and ask further questions based on the contents.

[0031] ●Explanation of language model learning function As described above, the present language model GM is stored in the information processing server 2. The server information processing unit 10 of the information processing server 2 according to this embodiment has a function of machine learning the present language model GM (hereinafter referred to as the "language model learning function"). The present language model GM learned by machine learning by the server information processing unit 10 corresponds to the "trained model." The language model learning function will be described below.

[0032] As described above, the language model GM is a large-scale language model (LLM). In this embodiment, machine learning of the language model GM means fine-tuning (post-learning). In other words, the server information processing unit 10 performs fine-tuning on a large-scale language model that has undergone sufficient pre-training, using training data. The language model GM is expected to be used to generate "questions to be asked by an assessor in an assessment." The machine learning by the server information processing unit 10 is performed with the aim of training (tuning) the language model GM so that it outputs highly accurate questions. The accuracy of a question refers to its consistency with the questions actually provided by the assessor in the interview; the higher the consistency, the higher the accuracy.

[0033] The machine learning performed by the server information processing unit 10 is divided into a basic learning process and a first applied learning process. Each process will be described below.

[0034] Basic learning process In the basic learning process, the server information processing unit 10 performs machine learning (fine tuning) of the present language model GM using basic learning data 20. Fig. 3 is a diagram showing information included in the basic learning data 20. As shown in Fig. 3, the basic learning data 20 includes assessment standard information 21 and functional safety standard information 22.

[0035] The assessment standard information 21 is information relating to the A-SPICE standard. The assessment standard information 21 includes, for example, definitions of terms used in A-SPICE, definitions relating to process reference models, and various definitions relating to processes. A-SPICE-related standards can be used as part of the assessment standard information 21. In this embodiment, the assessment standard information includes information indicating the confirmation theme for each evaluation unit.

[0036] The functional safety standard information 22 is information about functional safety standards related to A-SPICE. Questions asked by the assessor in the unit interview may include matters related to functional safety standards. In addition, functional safety standards may be used and referenced in the context information 26 described below. Based on the above, the functional safety standard information 22 is included in the basic training data 20.

[0037] In the basic learning process, the server information processing unit 10 performs machine learning (fine tuning) on ​​the language model GM using basic learning data so that the language model GM learns the A-SPICE standard and the functional safety standard. When machine learning the language model GM, the basic learning data is converted into data in a format suitable for machine learning based on the learning purpose and the specifications of the language model GM. When machine learning the language model GM (and other models as well), the learning data used for machine learning is converted into data suitable for machine learning based on the learning purpose, even if not specifically explained below.

[0038] The information included in the basic training data 20 is not limited to the information exemplified in this embodiment. The basic training data 20 may include information on various standards, information used as a dictionary, and other information related to background knowledge.

[0039] ○First applied learning process Next, the first applied learning process will be described. In the first applied learning process, assessment-related learning data 24 is prepared for each assessment conducted in the past. FIG. 4 is a diagram showing information contained in the assessment-related learning data 24. As shown in FIG. 4, the assessment-related learning data 24 includes a question learning dataset 25 and context information 26.

[0040] FIG. 5 is a diagram illustrating an example of the contents of the question learning dataset 25 in a manner suitable for explanation. The question learning dataset 25 corresponding to one assessment includes, for each combination of assessment unit ID, project feature, and confirmation theme, appropriate (correct) questions (sentences indicating questions) to be asked by the assessor in that assessment. Each question reflects the "question context" of that assessment. The question context will be described later. Each question may be a question actually provided by the assessor in that assessment (including an adjustment of this question), or may be an appropriate question created.

[0041] For example, in record R1 in Figure 5, the evaluation target ID: SWE.2.BP1, project characteristics: normal, confirmation theme: goal setting, and a question are associated. The question in this record R1 indicates an appropriate (correct) question when the confirmation theme is "goal setting" in the unit interview related to "SWE.2.BP1" of the corresponding project (project characteristics are normal). The corresponding project refers to the project in which the assessment that served as the basis for the assessment-related learning data 24 to which the question learning dataset 25 belongs was conducted. Also, for example, in record R2 in Figure 5, the evaluation target ID: MAN.3.BP1, project characteristics: normal, confirmation theme: project goal, and a question are associated. The question in this record R2 indicates an appropriate (correct) question when the confirmation theme is "project goal" in the unit interview related to "MAN.3.BP1" of the corresponding project (project characteristics are normal). Note that the project feature values ​​of all records belonging to a given question learning dataset 25 are the same. This is because the project corresponding to the question learning dataset 25 is the same.

[0042] The question learning dataset 25 functions as training data for supervised learning in fine-tuning. The question learning dataset 25 is included in the assessment-related training data 24 for one of the purposes of training the language model GM to output appropriate (correct) questions according to a combination of assessment unit ID, project features, and confirmation theme. The question learning dataset 25 is also included in the assessment-related training data 24 for one of the purposes of training the language model GM to correctly estimate project features from context information 26 (described below).

[0043] 4, the context information 26 includes an assessment plan 27 (assessor use information), a project plan 28 (assessee use information), a deliverable unit 29, interview conversation information 30, and an assessment report 31. Hereinafter, the assessment plan 27, the project plan 28, the deliverable unit 29, the interview conversation information 30, and the assessment report 31 will be collectively referred to as "individual context information."

[0044] The context information 26 included in a given piece of assessment-related training data 24 functions as the context of a question recorded in a question learning dataset 25 included in that piece of assessment-related training data 24. The context of a question refers to the background that led to the question being provided by the assessor, or the background in which the assessor set the content of the question. In this embodiment, "background" is a concept that includes context, situation, premise, or "specific individual circumstances related to the assessment / project." As will be described later, when training the language model GM using the assessment-related training data 24, each piece of context information 26 is used as the context of the corresponding question learning dataset 25. Below, the content of each piece of individual context information will be explained, along with the reason why the individual context information functions as the context of a question.

[0045] The assessment plan 27 is information that records the plan for the corresponding assessment. The assessor creates the assessment plan 27 before the assessment and uses the assessment plan 27 to carry out the assessment. The assessment plan 27 usually records information that indicates the content of the project to be assessed and the rules that will be followed in the assessment. For example, information on the following items is recorded in the assessment plan 27: Project Background -Description of the development Life cycle stages ·Number of personnel ·Safety characteristics Tools used Security properties Assessment model Assessment Guidelines

[0046] The project background is, for example, an overview of the project, the project's objectives, the future outlook for the project, or the reason the project was started. The description of the development product is a description of the development product to be developed by the system development. The number of personnel is the number of personnel required for system development. The safety characteristics are the safety standards to which the system development or development product complies. The tools used are the tools (e.g., platform or programming language) used in system development. The security characteristics are the security standards to which the system development or development product complies. The assessment model is the assessment standard (including version information) to which the assessment complies. The assessment guidelines are the guidelines to which the assessment complies. In the above example, the project background, description of the development product, life cycle stage, number of personnel, safety characteristics, security characteristics, and tools used correspond to "information that indicates the content of the project," and the assessment model and assessment guidelines correspond to the rules to which the assessment complies.

[0047] The assessment plan 27 records information that indicates or allows for the inference of the context of a question. For example, the "information indicating the content of the project to be assessed" (project background, description of the development, lifecycle stage, number of personnel, safety characteristics, security characteristics, and tools used) included in the assessment plan 27 each contain information that indicates the specific circumstances of the project, and therefore can be said to indicate or allow for the inference of the context of a question. Similarly, the "rules to be followed in the assessment" (assessment model and assessment guidelines) contain information that indicates the assumptions of the assessment, and therefore can be said to indicate or allow for the inference of the context of a question. Therefore, the information recorded in the assessment plan 27 included in one assessment-related learning data 24 functions as the context of the questions recorded in the question learning dataset 25 included in that one assessment-related learning data 24.

[0048] The assessment plan 27 is included in the assessment-compatible training data 24 so that the question training data set 25 is trained while reflecting the context of the question, thereby improving the accuracy of the questions output by the language model GM.

[0049] The assessment plan 27 includes information that enables the derivation of the project characteristics of the corresponding project. For example, the value of the item: development period in the assessment plan 27 can be used to determine the "length of the development period," and the values ​​of the item: safety characteristics and item: security characteristics can be used to determine the "specific grade of functional safety."

[0050] The project plan 28 is information that records the plan for the corresponding project. The assessee creates the project plan 28 and uses it to carry out the project. The project plan 28 records, for example, the project outline (purpose, scope, development man-hours, etc.), project configuration (life cycle model, tailoring status, organizational structure, etc.), project management (management goals, list of deliverables, development schedule, test plan, etc.), and assessment plan.

[0051] The information recorded in the project plan 28 serves as context for the questions recorded in the corresponding question learning dataset 25 for similar reasons as the information in the assessment plan 27. The project plan 28 is also included in the assessment-enabled learning data 24 for similar purposes as the assessment plan 27.

[0052] A deliverable unit 29 is information composed of deliverables created during the system development process in a project. A deliverable unit 29 typically includes multiple deliverables. In particular, a deliverable unit 29 includes deliverables that the assessor has the assessee submit in the unit interview. As described above, in the unit interview, the assessor appropriately has the assessor submit deliverables, checks their contents, and asks questions based on those contents. For this reason, the information recorded in the deliverables includes information that indicates or infers the background to the questions asked. Therefore, the information recorded in the deliverable unit 29 functions as the context for the questions recorded in the corresponding question learning dataset 25. The deliverable unit 29 is included in the assessment-related learning data 24 for the same purpose as the assessment plan 27.

[0053] The interview conversation information 30 is information indicating the actual conversation that took place between the assessor and the assessee in the unit interview conducted in the corresponding assessment. For each unit interview, the interview conversation information 30 includes an assessment unit ID and "conversation information recording the conversation between the assessor and the assessee in the unit interview." The conversation information records a clear distinction between the assessor's statements and the assessee's statements. Naturally, the content of the conversation between the assessor and the assessee in the unit interview will reflect the individual and specific circumstances of the assessment and project. Therefore, the interview conversation information 30 functions as the context for the questions recorded in the corresponding question-based learning dataset 25. The interview conversation information 30 is included in the assessment-related learning data 24 for the same purpose as the assessment plan 27.

[0054] The assessment report 31 is a report on the assessment prepared by the assessor after the assessment. The assessment report 31 includes information indicating the process maturity assessed by the assessor for each process. The assessment report 31 also records the assessor's findings for each assessment unit. The findings include information obtained by the assessor about the assessment unit, information indicating why and how the assessment was made, and any points the assessor considers to be important. The assessment report 31 is a medium prepared based on the assessor's actual experience, and records information that indicates or allows one to infer the background of the questions posed in the unit interview. Therefore, the assessment report 31 functions as the context for the questions recorded in the question-based learning dataset 25. The assessment report 31 is included in the assessment-related learning data 24 for the same purpose as the assessment plan 27.

[0055] As described above, in this embodiment, assessment-related training data 24 is prepared for each assessment conducted in the past. Then, in the first applied learning process, the server information processing unit 10 acquires each piece of assessment-related training data 24. Next, for each piece of assessment-related training data 24, the server information processing unit 10 performs machine learning (fine-tuning) of the present language model GM using the context information 26 as context and the question learning dataset 25 as training data. That is, in the machine learning, the server information processing unit 10 uses the context information 26 of one piece of assessment-related training data 24 as information indicating the context of a question recorded in the question learning dataset 25 of that one piece of assessment-related training data 24 to perform machine learning of the present language model GM.

[0056] The process in which the server information processing unit 10 machine-learns the language model GM using assessment-compatible learning data 24 including context information 26 corresponds to "a process in which the language model constructs a state in which questions can be generated by reflecting context information including assessor usage information as context" and "a process in which the language model is trained using learning data including context information as context."

[0057] As described above, the server information processing unit 10 trains the language model GM using assessment-related training data 24, which includes context information 26 as context. Therefore, the language model GM is trained while reflecting the context. Questions generated by the language model GM reflect the project and assessment entities, and are expected to closely match questions actually posed by assessors in interviews and therefore to be highly accurate. In particular, the assessment-related training data 24 includes a question training dataset 25 corresponding to a past assessment and an assessment plan 27 (assessor usage information) used by the assessor in that assessment as context. This assessment plan 27 includes information about the project and the assessment necessary for the appropriate execution of the assessment. Therefore, the questions generated by the language model GM are expected to be more accurate. Furthermore, in this embodiment, the context information 26 includes not only the assessment plan 27 but also a project plan 28, a deliverable unit 29, interview conversation information 30, and an assessment report 31. Therefore, it is expected that the accuracy of questions generated by the present language model GM will be improved by including multifaceted and multifaceted information that functions as context in the assessment-compatible training data 24.

[0058] ●Explanation of the pseudo-assessment service The server information processing unit 10 has a function of providing a pseudo-assessment service using the present language model GM. The pseudo-assessment service is a service that provides the assessee with a chat room 33 in which to chat with a "pseudo assessor using an AI robot" (hereinafter referred to as the "pseudo-assessor") and realizes a pseudo-unit interview in this chat room 33. The assessee can receive questions from the pseudo-assessor in the chat room 33 and answer them, thereby experiencing a pseudo-unit interview. The pseudo-assessment service can be used by assessees who are currently executing or have already completed a project and who plan to undergo an assessment in the future. However, users of the pseudo-assessment service are not limited to assessees. An assessor may also use the pseudo-assessment service. The operation of the assessment-related system 1 in the pseudo-assessment service will be described below.

[0059] Figure 6 is a diagram illustrating the terminal information processing unit 13, the server information processing unit 10, and the language model GM in a manner suitable for explanation, in order to explain the operation of the assessment-related system 1 in the pseudo-assessment service. Hereinafter, it is assumed that the assessee is seeking to receive the pseudo-assessment service for a specific assessment unit of a project he or she is carrying out. It is also assumed that the terminal storage unit 17 of the assessee terminal 3 stores the project plan 28 of the assessee's project and files of the necessary deliverables.

[0060] When using the pseudo assessment service, the person being assessed first launches the dedicated application AP. After the dedicated application AP is launched, the terminal information processing unit 13 basically executes various processes using the functions of the dedicated application AP. The terminal information processing unit 13 also displays various screens and executes other processes as appropriate in cooperation with the server information processing unit 10. The person being assessed performs a predetermined operation on the user interface provided by the functions of the dedicated application AP to instruct the start of the provision of the pseudo assessment service.

[0061] In response to an instruction to start providing the pseudo-assessment service, the terminal information processing unit 13 displays an evaluation unit selection screen on the terminal display unit 15. The evaluation unit selection screen displays information indicating the evaluation units in a selectable state. The information indicating the evaluation units is, for example, a combination of an evaluation unit ID and a brief description of the evaluation unit. The person being appraised selects the evaluation unit for which they wish to undergo a pseudo-unit interview. Once the evaluation unit is selected by the person being appraised, the terminal information processing unit 13 displays a project plan upload screen on the terminal display unit 15. The project plan upload screen is a screen for uploading a file of the project plan 28 to the information processing server 2. The screen displays an object for specifying the file to be uploaded and for instructing the upload of the specified file. The person being appraised uses the object to specify the project plan 28 of the project to be the target of the pseudo-assessment service and instructs the upload of the plan. In response to the instruction, the terminal information processing unit 13 transmits (uploads) the evaluation unit ID of the evaluation unit selected by the person being appraised and the project plan 28 to the server information processing unit 10 (step SA1 in FIG. 6). Hereinafter, the project plan 28 uploaded to the information processing server 2 in the pseudo assessment service will be referred to as the "target project plan."

[0062] In response to an instruction from the assessee, the terminal information processing unit 13 further displays a chat room 33 on the terminal display unit 15. Fig. 7 is a diagram showing an example of the chat room 33. As shown in Fig. 7, the chat room 33 is provided with a message input field 34 for inputting a message and an upload object 35 for uploading a file.

[0063] The server information processing unit 10 receives and acquires the evaluation unit ID and the target project plan. Based on the evaluation unit ID and the target project plan, the server information processing unit 10 generates a prompt (hereinafter referred to as the "initial prompt") that requests the language model GM to generate the first question. The initial prompt includes the evaluation unit ID and the contents of the target project plan. The initial prompt is a prompt that requests the generation of the first question that the assessor should provide in the unit interview of the evaluation unit indicated by the evaluation unit ID in the assessment of the project indicated by the target project plan. The server information processing unit 10 generates the prompt using a program having a function of generating a prompt.

[0064] An example of an initial prompt is shown below. However, the primary purpose of the example of the initial prompt shown below is to show an example of how the evaluation target ID and project plan 28 are reflected in the initial prompt, and the appropriateness of the prompt as such is ignored. The specific content of the prompt should be set based on prompt engineering techniques so that an appropriate response is obtained. The above points also apply to other prompts.

[0065] ***Beginning of prompt*** You are an assessor who will assess the assessee in accordance with A-SPICE. The project to be assessed is as shown in the [Project Plan] below. Process: SWE.2 Base Practice: Answer the first question you provide to the assessor in your interview about BP1. [Project Plan] 1. Introduction ○○ Co., Ltd. is... 2. Project Overview 2-1. This project is... … 6. Project Management 6-1. Management Objectives and Priorities The management goal is... … ***END OF PROMPTS***

[0066] After generating the first prompt, the server information processing unit 10 outputs (provides) the first prompt to the language model GM (step SA2). The language model GM receives the first prompt. The language model GM generates a question (answer sentence) based on the first prompt. Hereinafter, a question generated by the language model GM is referred to as a "model question." In particular, the first question generated based on the first prompt is referred to as an "first question." The language model GM outputs the first question to the server information processing unit 10 (step SA3). The server information processing unit 10 acquires the first question. The server information processing unit 10 transmits the first question as a message to the terminal information processing unit 13 (step SA4). Note that the server information processing unit 10 may be configured to adjust / modify the content of the first question before transmitting the first question 10 to the terminal information processing unit 13. This adjustment / modification may, for example, involve adding a question mark to the end of the word.

[0067] The terminal information processing unit 13 receives the initial question and displays it in the chat room 33 in accordance with the rules. The displayed question is treated as a question asked by the pseudo assessor. Figure 8 shows the initial question displayed in the chat room 33. The assessee refers to the initial question and answers it based on the substance, situation, plan, background, etc. of their own project. The assessee (respondent) answers the model-side question by entering a sentence indicating the answer in the message input field 34 and confirming the input. Hereinafter, the answer by the assessee to the model-side question (= the sentence entered in the message input field 34) is referred to as the "respondent's answer." When the respondent's answer is entered in the message input field 34, the terminal information processing unit 13 displays the respondent's answer in the chat room 33 in accordance with the rules. Figure 9 shows the respondent's answer to the initial question displayed in the chat room 33.

[0068] Furthermore, the terminal information processing unit 13 transmits the answerer's answer to the server information processing unit 10 (step SA5). The server information processing unit 10 receives and acquires the answerer's answer. Next, the terminal information processing unit 13 generates a prompt that includes the answerer's answer and requests that the next question be generated reflecting the answerer's answer. Hereinafter, the prompt generated by the terminal information processing unit 13 in response to the answerer's answer is referred to as a "re-question request prompt." The server information processing unit 10 outputs (provides) the generated re-question request prompt to the language model GM (step SA6). The language model GM inputs the re-question request prompt. The language model GM generates a model question based on the re-question request prompt and outputs it to the server information processing unit 10 (step SA7). The server information processing unit 10 acquires the model question. The server information processing unit 10 transmits the model question as a message to the terminal information processing unit 13 (step SA8). The terminal information processing unit 13 receives the model question and displays it in the chat room 33 according to the rules. Thereafter, the conversation between the assessee and the pseudo assessor is repeated until the language model GM outputs a response indicating that the unit interview is to be ended.

[0069] Here, based on interview feature F3, it is assumed that the model question includes a request for the submission of a specific type of deliverable. Hereinafter, a question including a request for the submission of a deliverable is referred to as a "submission request question." When the submission request question is displayed as a message in chat room 33, the assessee uploads a file of the corresponding specific type of deliverable using upload object 35. Once the file of the specific type of deliverable is uploaded, server information processing unit 10 acquires the file. Next, server information processing unit 10 generates a re-question request prompt that includes the contents of the file and requests that the next question be generated based on the contents of the file. Server information processing unit 10 outputs the generated re-question request prompt to the language model GM. The language model GM generates a model question based on the re-question request prompt and outputs it to the server information processing unit 10. As a result of the above processing, the language model GM generates a highly accurate question that reflects interview feature F3 and takes into account the actual assessment situation.

[0070] In this way, the server information processing unit 10 repeatedly executes the process of "requesting the present language model GM to generate model-side questions, providing the generated model-side questions to the assessee, obtaining any respondent answers from the assessee, and again requesting the present language model GM to generate model-side questions." This realizes a pseudo-unit interview between the assessee and the pseudo assessor.

[0071] As explained above, the server information processing unit 10 according to this embodiment has a function of constructing a state in which the language model GM can generate questions by reflecting, as context, context information including the assessment plan 27 (assessor usage information) used by an assessor when assessing a past project according to a predetermined A-SPICE. This provides the following advantages. Namely, questions generated by the language model GM reflect the substance of the project and the substance of the assessment, and are expected to be highly consistent with questions actually posed by assessors in interviews, and therefore highly accurate. Based on the above, this embodiment enables effective preparation for interviews conducted in assessments.

[0072] Second Embodiment Next, a second embodiment will be described. In the following description of the second embodiment, the same elements as those in the first embodiment will be given the same reference numerals, and detailed description thereof will be omitted.

[0073] Fig. 10 is a block diagram showing an example of the functional configuration of an assessment-related system 1A according to this embodiment. In Fig. 10, an information processing server 2A corresponds to the information processing server 2 of the first embodiment, and a server information processing unit 10A corresponds to the server information processing unit 10 of the first embodiment. As shown in Fig. 10, a knowledge database 37 is stored in the server storage unit 12 of the information processing server 2A.

[0074] ●Second applied learning process The server information processing unit 10A according to this embodiment executes a second applied learning process instead of the first applied learning process according to the first embodiment. The second applied learning process will be described below.

[0075] In the second applied learning process, a comprehensive question learning dataset 38 is prepared as learning data instead of the assessment-related learning data 24 (group) according to the first embodiment. FIG. 11 is a diagram showing the comprehensive question learning dataset 38. As shown in FIG. 11, the comprehensive question learning dataset 38 includes appropriate (correct) questions for each combination of evaluation unit ID, project feature, and confirmation theme. In the second applied learning process, the server information processing unit 10A uses the comprehensive question learning dataset 38 as training data to perform machine learning (fine-tuning) of the language model GM. As such, in this embodiment, context information 26 is not used in the machine learning of the language model GM.

[0076] ●Database registration process The server information processing unit 10A according to this embodiment has a function of executing a database registration process, which will be described below.

[0077] During the database registration process, context information 26 is prepared for each of the assessments conducted in the past. As described above, the context information 26 is composed of an assessment plan 27, a project plan 28, a deliverable unit 29, interview conversation information 30, and an assessment report 31. The server information processing unit 10A registers each piece of context information 26 in the knowledge database 37. The server information processing unit 10A registers each piece of context information 26 in the knowledge database 37 in a manner that allows the context information 26 registered in the knowledge database 37 to be searched when generating a prompt. For example, the server information processing unit 10A executes the following process. That is, the knowledge database 37 is composed of a vector database. Then, for each piece of context information 26, the server information processing unit 10A vectorizes the context information 26 (the context information may be divided into chunks and vectorized during vectorization), and registers the vector information of the context information 26 together with the original text data in the knowledge database 37 according to the format.

[0078] The process in which the server information processing unit 10A registers the context information 26 in the knowledge database 37 corresponds to "a process in which the language model constructs a state in which it can generate a question by reflecting the context information including the assessor usage information as context" and "a process in which it constructs a state in which it can use the context information registered in the database when the language model generates a question."

[0079] The database registration process corresponds to a process for implementing RAG (Retrieval-Augmented Generation). Specific aspects of the database registration process are not limited to the exemplified aspects. In other words, the database registration process may be any process that establishes a state in which the context information 26 registered in the knowledge database 37 can be searched when a query generation request is made to the language model GM.

[0080] ● Pseudo assessment service The assessment-related system 1A according to this embodiment executes the following processing in the pseudo-assessment service. FIG. 12 is a diagram used to explain the operation of the assessment-related system 1A in the pseudo-assessment service. FIG. 12 illustrates the terminal information processing unit 13, the server information processing unit 10A, the present language model GM, and the knowledge database 37 in a manner suitable for explanation. Hereinafter, with reference to FIG. 12, an example of the operation of the assessment-related system 1A from the start of provision of the pseudo-assessment service until the second model-side question is provided to the assessee will be explained. However, in this embodiment, the chat room 33 is displayed in the same manner as in the first embodiment, and therefore a description of this display will be omitted.

[0081] The appraisee starts the dedicated app AP and performs a predetermined operation to select an evaluation unit and instruct the upload of the project plan 28 (project-related information). In response to the instruction, the terminal information processing unit 13 transmits the evaluation unit ID of the evaluation unit selected by the appraisee and the project plan 28 to the server information processing unit 10A (step SB1 in FIG. 12).

[0082] The server information processing unit 10A receives and acquires the evaluation unit ID and the target project plan. The server information processing unit 10A accesses the knowledge database 37 (step SB2). Based on the target project plan, the server information processing unit 10A searches and acquires context information 26 from the knowledge database 37 that the language model GM should reflect as context when generating model-side questions (step SB3). The server information processing unit 10A performs the search, for example, using the following method. That is, the server information processing unit 10A vectorizes the target project plan. Next, the server information processing unit 10A derives the similarity between the vector information of the target project plan and each vector information of the context information 26 registered in the knowledge database 37 using a predetermined algorithm, and identifies the context information 26 with the highest similarity (it may be multiple context information 26 with similarities above a certain level). The server information processing unit 10A performs the search using the above method.

[0083] However, the search method by the server information processing unit 10A is not limited to the exemplified method. That is, the search method may be any method that discovers the context information 26 that the language model GM should reflect as context when generating a model-side question. For example, the following method may be used. That is, in the database registration process, the server information processing unit 10A registers, for each piece of context information 26 in the knowledge database 37, the vector information of the project plan 28 included in the context information 26, in association with the context information 26 as a search key (meta information). The server information processing unit 10A then derives the similarity between the vector information of the target project plan and the search key (vector information of the project plan 28) in the knowledge database 37, and identifies the context information 26 corresponding to the search key with the highest similarity (this may be multiple pieces of context information 26 with a similarity equal to or greater than a certain level). For example, the above method may be used. The search key is not limited to the vector information of the project plan 28, and may be key information that contributes to the discovery of appropriate context information 26. For example, the search key may be vector information of the assessment plan 27 or vector information of a combination of the project plan 28 and the assessment plan 27.

[0084] Hereinafter, the context information 26 acquired in step SB3 will be referred to as "extracted context information." After acquiring the context information 26, the server information processing unit 10A generates an initial prompt. In this embodiment, the initial prompt includes the evaluation unit ID, the contents of the target project plan, and the contents of the extracted context information. The initial prompt is a prompt that requests the generation of the first question that the assessor should ask in the unit interview of the evaluation unit indicated by the evaluation unit ID, using the extracted context information as context, in the assessment of the project indicated by the target project plan. The server information processing unit 10A generates the prompt using a program that has the function of generating a prompt. An example of the initial prompt is as follows:

[0085] ***Beginning of prompt*** You are an assessor who will assess the assessee in accordance with A-SPICE. The project to be assessed is as shown in the [Project Plan] below. Process: SWE.2 Base Practice: Answer the initial questions you provide to the assessor in your interview about BP1. Use the following [Context Information] as context when generating your questions. [Project Plan] 1. Introduction ○○ Co., Ltd. is... … [Context Information] Assessment plan 1.Business background… … Project Plan 1. Introduction … ·Deliverables Unit 1. Outline design document… … Interview conversation information 1.SWE.1… … Assessment Report 1. Process Maturity by Process… … ***END OF PROMPTS***

[0086] After generating the first prompt, the server information processing unit 10A outputs (provides) the first prompt to the present language model GM (step SB4). The present language model GM generates a first question (answer sentence) based on the first prompt. The present language model GM outputs the first question to the server information processing unit 10A (step SB5). The server information processing unit 10A acquires the first question. The server information processing unit 10A transmits the first question as a message to the terminal information processing unit 13 (step SB6).

[0087] The terminal information processing unit 13 receives the initial question and displays it in the chat room 33. The assessee (respondent) inputs the respondent's answer in the message input field 34. The terminal information processing unit 13 transmits the respondent's answer to the server information processing unit 10A (step SB7). The server information processing unit 10A receives and acquires the respondent's answer. The server information processing unit 10A generates a re-question request prompt that reflects the respondent's answer. The server information processing unit 10A outputs (provides) the generated re-question request prompt to the language model GM (step SB8). The language model GM receives the re-question request prompt and generates a model question based on the re-question request prompt. The language model GM outputs the model question to the server information processing unit 10A (step SB9). The server information processing unit 10A acquires the model question. The server information processing unit 10A transmits the model question as a message to the terminal information processing unit 13 (step SB10). The terminal information processing unit 13 receives the model-side question and displays it in the chat room 33 according to the rules. After that, the conversation between the assessee and the assessor is repeated until the language model GM outputs a response indicating that the unit interview is to be ended.

[0088] The configuration of the second embodiment also achieves the same effects as the first embodiment. That is, the language model GM generates model questions using context information 26 (including, in particular, the assessment plan 27) as context. Therefore, the model questions reflect the substance of the project and the substance of the assessment, and are expected to be highly consistent with the questions actually asked by the assessor in the interview, and therefore highly accurate. Based on the above, the present invention enables the assessee to make effective preparations for the interview conducted in the assessment.

[0089] In this embodiment, the server information processing unit 10A may be configured to obtain context information 26 from the knowledge database 37 and provide it to the language model GM each time it generates a re-question prompt. Alternatively, the server information processing unit 10A may be configured to output the context information to the language model GM separately from the prompt, rather than recording it in the prompt.

[0090] Although the above describes various embodiments of the present invention, these embodiments merely illustrate examples of specific embodiments for carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereby. In other words, the present invention can be embodied in various forms without departing from the gist or main features thereof. Below, variations of the above embodiments are presented. However, if multiple variations described below can be applied in combination, they may be applied in combination.

[0091] For example, in each of the above embodiments, the pseudo-assessment service may be configured such that after a simulated interview for an evaluation unit is completed, the server information processing unit 10 automatically evaluates all or part of the project. For example, the server information processing unit 10 may be configured to derive an evaluation value for the evaluation unit based on the conversation held in the simulated interview. In this configuration, the server information processing unit 10 executes the following process, for example. For example, the server information processing unit 10 determines whether or not tasks have been accomplished for each confirmation theme of the evaluation unit based on the content of the conversation. Then, the server information processing unit 10 derives an evaluation value for the evaluation unit based on the achievement rate of the tasks for each confirmation theme. In this configuration, the server information processing unit 10 may be configured to evaluate process maturity based on the evaluation values ​​derived for multiple evaluation units.

[0092] In the first embodiment, the present language model GM is stored in the server storage unit 12 of the information processing server 2. In this regard, the present language model GM trained by the information processing server 2 may be stored in an external device different from the information processing server 2. Fig. 13(A) shows an example of the configuration of the assessment-related system 1 when the present language model GM is stored in a language model server 40 different from the information processing server 2. Note that in Fig. 13(A), the information processing server 2 and the language model server 40 are connected via a network N, but the connection form between the information processing server 2 and the language model server 40 is not limited. The information processing server 2 and the language model server 40 may be connected via a LAN, or may be connected via a dedicated line.

[0093] In the second embodiment, the language model GM and knowledge database 37 are stored in the server storage unit 12 of the information processing server 2A. In this regard, the language model GM and knowledge database 37 may be stored in an external device different from the information processing server 2. Fig. 13(B) shows an example of the configuration of the assessment-related system 1A in which the language model GM is stored in a language model server 40 different from the information processing server 2, and the knowledge database 37 is stored in a database server 41 different from the information processing server 2. Below, a brief description will be given of the operation of the assessment-related system 1A for the pseudo-assessment service in the example of Fig. 13(B).

[0094] FIG. 14 is a diagram illustrating an example of the operation of the assessment-related system 1A up to the time the initial question is provided to the evaluator in the example of FIG. 13 (B). In response to the assessee's selection of an assessment unit and an instruction to upload the target project plan, the terminal information processing unit 13 transmits the assessment unit ID and the target project plan to the server information processing unit 10A (step SC1 in FIG. 14). The server information processing unit 10A receives and acquires the assessment unit ID and the target project plan. The server information processing unit 10A accesses the knowledge database 37 of the database server 41 (step SC2). The server information processing unit 10A searches and acquires context information 26 from the knowledge database 37 based on the target project plan (step SC3). An initial prompt is generated reflecting the target project plan and the extracted context information. The server information processing unit 10A transmits (provides) the initial prompt to the language model server 40 (step SC4). The language model server 40 generates an initial question (answer sentence) based on the initial prompt using the function of the language model GM. The language model server 40 transmits the initial question to the server information processing unit 10 (step SC5). The server information processing unit 10 receives and acquires the initial question. The server information processing unit 10 transmits the initial question as a message to the terminal information processing unit 13 (step SC6). This concludes a brief description of the operation of the assessment-related system 1A.

[0095] In addition, in each of the above embodiments, the user of the pseudo-assessment service is the person being assessed, but the user is not limited to the person being assessed. The pseudo-assessment service can also be used by assessors. In particular, by assessors using the pseudo-assessment service at the assessment site, it is possible to shorten the assessment time and prevent mistakes.

[0096] In addition, in each of the above embodiments, the information processing server 2 provided the pseudo-assessment service. In other words, the information processing server 2 functioned as a control system for providing the pseudo-assessment service. In this regard, the system providing the pseudo-assessment service may be a system different from the information processing server 2. In this case, the system functions as a control system.

[0097] Furthermore, in each of the above embodiments, a part or all of the processes described as being executed by the functions of the dedicated application AP may be configured to be executed by the functions of a general-purpose browser.

[0098] In each of the above embodiments, the assessment standard to which the assessment conforms is not limited to A-SPICE, but may be, for example, CMMI.

[0099] In the above embodiments, the base practices and generic practices are the "evaluation units." However, the evaluation units are not limited to those exemplified, and may be any units that are the subject of interviews.

[0100] In the above embodiments, the project plan 28 functions as project-related information (information indicating the project) in the simulated assessment service. However, the project-related information is not limited to the project plan. For example, the project-related information may be information explicitly entered by the assessee as information indicating the project.

[0101] In addition, in each of the above embodiments, an example was given in which the present language model GM is used in a pseudo-assessment service. However, the method of using the present language model GM is not limited to the method exemplified. Because the present language model GM is an LLM, it is highly versatile and can be used in various assessment-related methods. For example, the present language model GM can be used to generate a list of questions to be asked by an assessor in an interview conducted in a situation that satisfies certain conditions.

[0102] In each of the above embodiments, the content of the context information 26 is not limited to the exemplified content. For example, the context information 26 may be composed of only the assessment plan 27, or may be composed of a combination of the assessment plan 27 and one or more other pieces of information.

[0103] In each of the above embodiments, the assessment use information is not limited to the assessment plan 27. In other words, the assessment use information may be any information that an assessor used when assessing a past project in accordance with a predetermined assessment standard. This also applies to the project plan 28.

[0104] In addition, in each of the above embodiments, the present language model GM may not be trained using the basic training data 20, but the basic training data 20 may be stored in the knowledge database 37 and provided to the present language model GM as needed.

[0105] In the above embodiments, the pseudo-assessment service is configured so that the assessee and the pseudo-assessor converse via chat. In this regard, the pseudo-assessment service may be configured so that the assessee and the pseudo-assessor converse via voice. In this configuration, the respondent's answers uttered by the assessee are appropriately converted into text using voice recognition technology or other technology. Furthermore, the model-side questions generated by the language model GM are appropriately output as voice using voice synthesis technology or other technology.

[0106] The functional blocks shown in the above embodiments can be realized by any hardware or by a combination of any hardware and any software, and are not limited to specific hardware.

[0107] The embodiments may also include the provision of a program executed by the information processing server 2 or the subject terminal 3, or the provision of the language model GM (trained model). The embodiments may also include the provision of a recording medium on which the program is recorded in a computer-readable manner. The recording medium may be a magnetic or optical recording medium or a semiconductor memory device. Specific examples include portable or fixed recording media such as flexible disks, hard disk drives (HDDs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray (registered trademark) discs, magneto-optical disks, flash memories, and card-type recording media.

[0108] Furthermore, the target functioning as an information processing system is not limited to the target exemplified in the above embodiment. Hereinafter, an example will be given focusing on the "function of constructing a state in which a language model can generate questions by reflecting, as context, context information including assessor usage information used by an assessor when assessing in a past project according to a predetermined assessment standard" (hereinafter referred to as the "first function"). In the above first embodiment, the information processing server 2 is configured to have the first function as shown in FIG. 15. In this configuration, the information processing server 2 functions as an information processing system, and the server information processing unit 10 functions as an information processing unit. On the other hand, as shown in FIG. 15(B), the information processing server 2 and an external device 42 that can communicate with it may be configured to realize the first function in cooperation. In this configuration, the information processing server 2 and the external device 42 function as an information processing system, and the server information processing unit 10 of the server information processing unit 10 and the external information processing unit 43 of the external device 42 function as an information processing unit. The above points also apply to a target functioning as a control system. [Explanation of symbols]

[0109] 1. 1A Assessment-related systems (information processing systems, control systems) 2. Information processing server (information processing system, control system) 10 Server information processing unit (information processing unit) 26 Context Information 27 Assessment Plan 28 Project Plan 29 Deliverable Unit (Deliverable) 30 Interview conversation information (conversation) 31 Assessment Report (Report) GM Language Model

Claims

1. an information processing unit that executes processing related to a language model used to generate questions to be asked by an assessor in accordance with a predetermined assessment standard for a system development project; The information processing unit The language model has a function of constructing a state in which the question can be generated by reflecting, as a context, context information including assessor usage information used when the assessor assessed in accordance with the predetermined assessment standard in a past project. An information processing system comprising:

2. The assessor use information includes information recorded in an assessment plan relating to a plan for assessment of the one project.

2. The information processing system according to claim 1, wherein:

3. The information recorded in the assessment plan includes information indicating the content of the one project.

3. The information processing system according to claim 2.

4. The context information includes information used by the assessee in carrying out the one project.

2. The information processing system according to claim 1, wherein:

5. The assessee use information includes information recorded in a project plan related to the plan of the one project.

5. The information processing system according to claim 4.

6. The context information includes the deliverables created by the assessee in the process of system development in the one project.

2. The information processing system according to claim 1, wherein:

7. The context information includes a conversation between the assessor and the assessee in an interview conducted by the assessor with the assessee in the one project.

2. The information processing system according to claim 1, wherein:

8. The context information includes information recorded in a report on the assessment prepared by the assessor after the assessment of the one project.

2. The information processing system according to claim 1, wherein:

9. The information processing unit has a function of training the language model using training data that includes the context information as a context.

9. The information processing system according to claim 1, wherein the information processing system comprises: a processor;

10. The information processing unit The language model has a function of registering the context information in a database and creating a state in which the context information registered in the database can be used when the language model generates the question.

9. The information processing system according to claim 1, wherein the information processing system comprises: a processor;

11. A control system capable of using the language model trained by the information processing system according to claim 9, obtaining project-related information indicative of the project; and generating a prompt requesting the assessor to generate questions to be asked in the assessment of the project indicated by the project-related information. A control system comprising:

12. a function of generating a prompt to request the next question to be generated by reflecting the answer given by the answerer to the question generated by the language model; 12. The control system of claim 11.

13. A control system capable of using the database in which the context information is registered by the information processing system according to claim 10, obtaining project-related information indicative of the project; a function of acquiring, from the database, the context information to be reflected as context in generating the question; and generating a prompt requesting the assessor to generate the question to be asked in the assessment of the project indicated by the project-related information, using the context information as a context. A control system comprising:

14. a function of generating a prompt to request the next question to be generated by reflecting the answer given by the answerer to the question generated by the language model; 14. The control system of claim 13.

15. A machine learning method using the information processing system according to claim 9, the information processing unit acquiring learning data including the context information as a context; the information processing unit performs machine learning on the language model using the training data. A machine learning method characterized by:

16. Machine learning is performed by the machine learning method according to claim 15. A trained model characterized by:

17. An information processing method by an information processing system that performs processing related to a language model used to generate questions to be asked by an assessor in accordance with a predetermined assessment standard for a system development project, comprising: an information processing unit of the information processing system acquiring context information including assessor usage information used by the assessor when assessing in accordance with the predetermined assessment standard in a past project; and a step in which the information processing unit of the information processing system constructs a state in which the question can be generated by reflecting, as a context, context information including the assessor usage information.

1. An information processing method comprising:

Images