Program, information processing method, and information processing apparatus

Abstract

We provide programs and other tools that enable the creation of reports, minutes, and other records more easily and accurately. [Solution] The program receives audio related to the event after the event has ended, provides the audio information (information based on the audio) to the language model 5, and causes the computer to execute a process to generate output text data consisting of multiple items including the date and circumstances. The program can also provide the language model 5 with prompts that define the content and format of the generated output text data, along with the audio information.

Description

Technical Field

[0001] The present invention relates to a program, an information processing method, and an information processing apparatus.

Background Art

[0002] In various operations, reports and minutes are created for the purpose of managing progress, schedules, etc. Patent Document 1 discloses a program for generating a summary text that summarizes text information included in one or a plurality of segment voice data.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Documents for reports, minutes, and other records (hereinafter referred to as "records, etc.") in various operations often need to be completed and submitted within a short time between operations, and the creation work is a burden on the person in charge. In addition, description errors are likely to occur due to the requirement to submit within a short time, and including the correction work can be a factor that further reduces work efficiency.

[0005] In one aspect of the present disclosure, a program or the like that can create reports, minutes, and other records, etc. more simply and accurately is provided.

Means for Solving the Problems

[0006] A program relating to one aspect of this disclosure causes a computer to perform a process that receives audio related to an event after the event has ended, provides audio information based on the audio to a language model, and generates output text data consisting of multiple items including the date and circumstances. [Effects of the Invention]

[0007] According to the program, etc., relating to one aspect of this disclosure, reports, minutes, and other records can be prepared more easily and accurately. [Brief explanation of the drawing]

[0008] [Figure 1] This is a schematic diagram of the record creation system according to Embodiment 1. [Figure 2] This is a block diagram showing an example of a server device configuration. [Figure 3] This is a block diagram showing an example configuration of a generation server device. [Figure 4] This is a block diagram showing an example configuration of a terminal device. [Figure 5] This is a flowchart illustrating an example of the process for creating record documents. [Figure 6] This figure shows an example of the item definition section included in a prompt. [Figure 7] This figure shows an example of output text data consisting of multiple items. [Figure 8] This figure shows an example of a registration information type selection screen. [Figure 9] This figure shows an example of an input screen. [Figure 10] This figure shows an example of a screen displaying converted text. [Figure 11] This figure shows an example of the output text data list screen. [Figure 12] This figure shows an example of the output screen. [Figure 13] This figure shows an example of an editing screen that displays all the text corresponding to a single item. [Figure 14]It is a diagram showing an example of configuring a system without providing a generation server device. [Figure 15] It is a schematic diagram of a recording document creation system according to Embodiment 2. [Figure 16] It is a block diagram showing a configuration example of a terminal device according to Embodiment 2.

Mode for Carrying Out the Invention

[0009] Hereinafter, an example of a program, an information processing method, and an information processing apparatus according to one aspect of the present disclosure will be described in detail based on the drawings. In the description, the same reference numerals are assigned to the same elements, and redundant descriptions are omitted as appropriate.

[0010] [Embodiment 1] FIG. 1 is a schematic diagram of a recording document creation system according to Embodiment 1. As shown in FIG. 1, the recording document creation system includes an information processing apparatus 1, an information processing apparatus 2, and a generation server apparatus 4, and each apparatus transmits and receives information via a network 3 such as the Internet. The generation server apparatus 4 may be communicatively connected so as to be able to utilize a language model 5 such as a large language model (LLM) stored inside or outside the generation server apparatus 4.

[0011] The information processing apparatus 1 is an information processing device that performs processing, storage, and transmission / reception of various information. The information processing apparatus 1 is, for example, an information processing device such as a server device, a personal computer, a tablet terminal, and a smartphone. The information processing apparatus 1 may be composed of a plurality of information processing devices, or may be configured as one of a plurality of virtual devices (virtual machines) configured within one information processing device. In this embodiment, for the sake of avoiding complicated explanations, the information processing apparatus 1 will be described by substituting it with the server apparatus 1. However, the information processing apparatus 1 is not limited to the server apparatus, and may be configured as any of the information processing devices as described above.

[0012] The information processing device 2 receives the information output from the server device 1 via the network 3, accepts the operations of the users of the information processing device 2, and transmits the information related to the operations to the server device 1 via the network 3. The information processing device 2 is an information processing device such as, for example, a personal computer, a smartphone, a mobile phone, a wearable device, and a tablet. In this embodiment, in order to avoid complication in explanation, the information processing device 2 is described by substituting it with the terminal device 2.

[0013] FIG. 2 is a block diagram showing a configuration example of the server device 1. The server device 1 may be an information processing device mainly composed of an electronic circuit using semiconductor circuit elements. The server device 1 includes a control unit 11, a communication unit 12, a reading unit 13, and a storage unit 14. Each configuration of the control unit 11, the communication unit 12, the reading unit 13, and the storage unit 14 is communicably connected to each other by a bus 19 or the like. Note that the server device 1 may have other configurations such as not having the reading unit 13.

[0014] The control unit 11 can be configured to have any one or more of processing devices such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), and a quantum processor. The control unit 11 can be configured to read and execute the program (or program product) P1 stored in the storage unit 14.

[0015] The communication unit 12 is a communication module for performing communication-related processing and can send and receive information with terminal devices 2, etc., via the network 3. The reading unit 13 can read portable storage media 1a such as CD (Compact Disc)-ROM, DVD (Digital Versatile Disc)-ROM, and USB (registered trademark) memory. The control unit 11 can read program P1 and / or data from the portable storage media 1a via the reading unit 13 and save it to the storage unit 14. The control unit 11 can also download program P1 from another computer via the network 3, etc., and save it to the storage unit 14.

[0016] The storage unit 14 may include volatile storage units such as RAM (Random Access Memory), and non-volatile storage units such as ROM (Read Only Memory), HDD (Hard Disk Drive), and flash memory. The control unit 11 can temporarily store programs and / or data read from the non-volatile storage unit or received from the communication unit 12 in the volatile storage unit for use. The storage unit 14 can store programs (or program products) P1 executed by the control unit 11, as well as data for prompts P2, which will be described later. The storage unit 14 can store multiple prompts P2.

[0017] Figure 3 is a block diagram showing an example configuration of the generation server device 4. The generation server device 4 can be an information processing device composed mainly of electronic circuits using semiconductor circuit elements. The generation server device 4 has a control unit 41, a communication unit 42, a reading unit 43, and a storage unit 44. The components of the control unit 41, communication unit 42, reading unit 43, and storage unit 44 are connected to each other so as to be able to communicate with one another by a bus 49 or the like. These components can be configured the same as the information processing device (server device) 1 in Figure 2, except that the program P1 and prompt P2 are not stored in the storage unit 44, so redundant explanations are omitted. Note that the generation server device 4 may have other configurations, such as not having a reading unit 43 for reading the portable storage medium 4a, as in the case of server device 1.

[0018] The generation server device 4 is an information processing device that handles language models 5, such as large language models (LLMs) like GPT (Generative Pre-trained Transformer, registered trademark), BERT (Bidirectional Encoder Representations from Transformer), and Gemini (registered trademark). The language models 5 may be stored in the memory unit 44 within the generation server device 4, or they may be stored in storage or the like that which is communicated with the generation server device 4.

[0019] The generation server device 4 receives input data such as images, audio, and text from the language model 5 and generates a response sentence. The language model 5 is a pre-trained machine learning model and can be a large-scale language model such as GPT (Generative Pre-trained Transformer) or BERT (Bidirectional Encoder Representations from Transformer), but other language models may also be used. The server device 1 may be connected to the generation server device 4 via the network 3, or it may be connected to the generation server device 4 directly or via a local network without using the network 3.

[0020] Figure 4 is a block diagram showing an example configuration of terminal device 2. Here, terminal device 2 can be an information processing device composed mainly of electronic circuits using semiconductor circuit elements. Terminal device 2 may have a control unit 21, a communication unit 22, a reading unit 23, a storage unit 24, a display unit 26, an input unit 27, and a microphone 28. The components of the control unit 21, communication unit 22, reading unit 23, storage unit 24, display unit 26, input unit 27, and microphone 28 may be connected to each other via a bus 29 or the like so that they can communicate with one another.

[0021] The control unit 21 can be configured to include one or more processing units such as a CPU (Central Processing Unit), MPU (Micro-Processing Unit), GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), DSP (Digital Signal Processor), and quantum processor. The control unit 21 can be configured to read and execute a program (or program product) stored in the storage unit 24, which will be described later.

[0022] The communication unit 22 is a communication module for performing communication-related processing and can send and receive information with the server device 1, etc., via the network 3. The reading unit 23 can read portable storage media 2a such as CD (Compact Disc)-ROM, DVD (Digital Versatile Disc)-ROM, and USB (registered trademark) memory. The control unit 21 can read programs and / or data from the portable storage media 2a via the reading unit 23 and save them to the storage unit 24. The control unit 21 can also download programs from other computers via the network 3, etc., and save them to the storage unit 24.

[0023] The storage unit 24 may include volatile storage devices such as RAM (Random Access Memory), and non-volatile storage devices such as ROM (Read Only Memory), HDD (Hard Disk Drive), and flash memory. The control unit 21 may temporarily store programs and / or data read from the non-volatile storage device or received from the communication unit 22 in the volatile storage device for high-speed reading and writing by the control unit 21.

[0024] The display unit 26 can be a liquid crystal display or an organic EL (Electro-Luminescence) display, etc. The input unit 27 can be an input device such as a keyboard, mouse, touch panel, and camera. The microphone 28 converts ambient sound around the terminal device 2 into digital data based on instructions from the control unit 21. The control unit 21 stores the converted digital data in the storage unit 24.

[0025] Figure 5 is a flowchart illustrating an example of the process for creating a record document. Server device 1 executes the process for creating a record document based on program P1, etc. As shown in this flowchart, server device 1 displays a registration information type selection screen 100 (Figure 8) for selecting the registration information type (step S101). The registration information type is the type of content of the record document that the user of terminal device 2 intends to record after the event has ended.

[0026] Here, "event" can refer to tasks such as meetings, interviews, and equipment maintenance. The registration information type can include, for example, reports on maintenance work, construction work, and other tasks, as well as records such as minutes of weekly or monthly meetings and other meetings. "Event" may also refer to situations where there is a need to order parts, reserve equipment, or make inquiries. Each registration information type may include common items such as date and time and circumstances, as well as items specific to each, such as selectively determined department names, meter values ​​for numerical input, and whether or not there are malfunctions.

[0027] The prompt P2 that specifies these items will be described in detail later in the explanation referring to Figure 6. For recording purposes, as described later, "event-related audio" from the user of terminal device 2 can be used. "Event-related audio" can include audio equivalent to meeting or interview minutes, audio equivalent to maintenance work reports, and audio such as parts orders, equipment reservations, and inquiries. Here, "audio" can include not only the audio itself but also audio data obtained by digitally converting the audio.

[0028] The server device 1, which executes program P1, can, for example, display multiple selectable registration information types on the display unit 26 of the terminal device 2, from among multiple registration information types pre-stored in the storage unit 14. Alternatively, the server device 1 may accept the selected registration information type when the user of the terminal device 2 selects one of the displayed registration information types.

[0029] Server device 1 determines whether it has accepted the selection of a registration information type on the registration information type selection screen 100 (step S102). If no registration information type is selected in step S102 (step S102: NO), the process in step S102 is repeated. If a registration information type is selected (step S102: YES), prompt P2 corresponding to the registration information type is selected (step S103).

[0030] The prompt P2 may be selected from several registration information types, such as "Visit Memo," "Maintenance Record," "Order Data Registration," "Provisional Reservation Application," and "Inquiry," as shown in Figure 8. Here, one or more prompts P2 corresponding to each registration information type may be stored in the storage unit 14 of the server device 1, as shown in Figure 2. Based on the instructions of program P1, the server device 1 can retrieve the prompt P2 corresponding to the selected registration information type from the storage unit 14.

[0031] When prompt P2 is selected, the server device 1, based on the instructions of program P1, sends an input screen 200 (form) (Figure 9) based on the content of the selected prompt P2 to the terminal device 2, and displays it on the display unit 26 of the terminal device 2 (step S104).

[0032] When the input screen 200 is displayed, terminal device 2 determines whether an instruction to start voice input has been received (step S105). If no instruction to start voice input is detected (step S105: NO), the process in step S105 is repeated. If an instruction to start voice input is detected (step S105: YES), terminal device 2 detects the voice via microphone 28 and converts it into voice data (step S106). Server device 1 receives the voice data via network 3. Hereinafter, if the terminal device 2 has been operated by voice input instructions from the stage of selecting the registration information type in step S102 or earlier, the process in step S105 may be omitted.

[0033] The input audio can be, for example, a reading of the following text. "Today, I had an appointment with Mr. X and visited ABC Corporation to meet with President Y. I was able to meet with President Y and Managing Director Z. Since it had only been about a week since the last proposal, I thought it might be difficult for them to make a decision on the contract, but the president gave his preliminary approval. I have already given them the contract, and I plan to receive the documents during my next visit on February 28th."

[0034] Server device 1 can store the received audio data in storage unit 14. Server device 1 can also convert the received audio data into audio information (e.g., audio-text data), which is information based on the audio. In this case, "audio information" can include the data of the received audio itself, the audio-text data obtained by converting the audio to text, and information on translations and other converted data thereof.

[0035] The speech-to-text conversion can be performed during speech input. In this case, the server device 1 may transmit the converted text information to the terminal device 2 and display the converted text display screen 300 (Figure 10) on the display unit 26 of the terminal device 2. In this embodiment, the speech-to-text conversion is performed during speech input, but it may also be performed after the completion of speech input. Furthermore, in this embodiment, the speech data is converted (e.g., to text), but text conversion is not required, and the speech data may be treated as speech information as is.

[0036] When voice input is finished, the user gives an instruction to end voice input using the input unit 27 or microphone 28. The control unit 21 of the terminal device 2 determines whether voice input has ended (step S107). If it determines that voice input has not ended (step S107: NO), the process in step S106 is repeated. If it determines that voice input has ended (step S107: YES), the server device 1 (receiving a notification of voice input completion from the terminal device 2) provides the voice information, which is information based on the received voice, to the language model 5 based on the instructions of program P1, and obtains and generates "output text data P3 consisting of multiple items including date and status" (step S108). The output text data P3 will be described in detail later in the explanation referring to Figure 7.

[0037] Here, server device 1 may provide voice information to language model 5 via generation server device 4 connected to network 3. The "output text data P3 consisting of multiple items including date and situation" may be output when prompt P2 given to language model 5 along with the voice information is specified to output text consisting of multiple items including date and situation. Alternatively, language model 5 itself may be configured to output the "output text data P3 consisting of multiple items including date and situation" without using prompt P2. Furthermore, for example, generation server device 4 may be configured to provide prompt P2 specified to output the "output text data P3 consisting of multiple items including date and situation".

[0038] Furthermore, in determining the end of voice input in step S107, terminal device 2 may terminate the voice input processing if it detects content in the voice information indicating that voice input has ended. This allows the user to end voice reception without touching terminal device 2. In addition, immediately following the voice reception termination process, server device 1 provides voice information such as voice data or voice-text data to language model 5 to generate output text data P3, allowing the user to view the output result screen 500 (Figure 12), described later, without touching terminal device 2. This enables the user to easily perform voice input, for example, when their hands are dirty at a work site.

[0039] Here, the process of providing the voice information in step S108 to the language model 5 to generate output text data P3 (hereinafter referred to as the "analysis process") can be performed as a background process by the server device 1 or the generation server device 4. In this case, the terminal device 2 can perform other operations, such as inputting voice of another registered information type, without waiting for the output text data P3 to be generated. Alternatively, the terminal device 2 may wait until it receives the output text data P3 instead of performing the process in the background.

[0040] In this way, the information processing device 1, based on the instructions of program P1, receives audio related to the event after the event has ended, provides the audio information, which is information based on the audio, to the language model 5, and generates output text data P3 consisting of multiple items including the date and circumstances. As a result, output text data P3 can be generated for multiple items included in a predetermined format such as a report, minutes, or other record, making the creation of such documents easier and more accurate.

[0041] Furthermore, the voice input time may be set to, for example, 5 minutes, 3 minutes, or 1 minute. This allows the input voice to be summarized in a short time, thereby shortening the input time. In addition, it is possible to create concise content that does not contain redundant information, which allows the language model 5 to convert it into output text data P3 with fewer errors and greater accuracy.

[0042] If the analysis process in step S108 is performed in the background, the server device 1 determines whether the display of the output text data list screen 400 (Figure 11) has been requested (step S109). If the display of the output text data list screen 400 has not been requested (step S109: NO), the process in step S109 is repeated. If the display of the output text data list screen 400 has been requested (step S109: YES), the output text data list screen 400 (Figure 11) is displayed on the terminal device 2 (S110).

[0043] The output text data list screen 400 may display a list of audio information for which the analysis processing in step S108 was requested. Alternatively, the server device 1 may display a list of audio information for which output text data P3 has been acquired on the output text data list screen 400. When displaying a list of audio information for which the analysis processing in step S107 was requested, the server device 1 may also indicate whether output text data P3 has been acquired (analyzed) or whether output text data P3 has not been acquired (analyzed). The server device 1 may also notify the terminal device 2 that the analysis is complete when the analysis processing is finished.

[0044] Next, on the output text data list screen 400, it is determined whether the audio information from which output text data P3 was obtained (analyzed) has been selected (step S111). If the audio information from which output text data P3 was obtained is not selected (step S111: NO), the process in step S111 is repeated. If the audio information from which output text data was obtained is selected (step S111: YES), the server device 1 displays the output result screen 500 on the terminal device 2 (step S112).

[0045] The output result screen 500 can be a screen that reflects the content of the output text data P3 on the input screen 200. For example, the server device 1 may send a form that reflects the content of the output text data P3 on the input screen 200 to the terminal device 2 and display it as the output result screen 500 on the display unit 26 of the terminal device 2. Alternatively, the server device 1 may send the output text data P3 as is to the terminal device 2, and the terminal device 2 may display the output result screen 500, which reflects the output text data P3 on the input screen 200, on the display unit 26.

[0046] The output result screen 500 will be described in detail later in the explanation referring to Figure 12. When the server receives an instruction to end the display of the output result screen 500, the record document creation process is terminated. The server device 1 may also display the output result screen 500 again by reading the output text data P3 or the data for displaying the output result screen 500 stored in the storage unit 14 in response to a request based on the operation of the user's terminal device 2.

[0047] Furthermore, if there is only one type of registration information, that is, if there is only one type of prompt P2 to select, or if the prompt P2 used by the server device 1 in the processing executed based on program P1 is fixed to one, then the processing in steps S101 to S103 can be skipped, and the process can start from step S104. Alternatively, the process in steps S101 to S104 can be skipped, and the process can start from step S105.

[0048] Figure 6 shows an example of the item definition section P21 included in prompt P2. Here, we will explain using the case where a record related to "Visit Memo" is selected as the registration information type. In Figure 6, the JSON (JavaScript® Object Notation) format is used, which allows each item and its value combination to be stored in an enumerated or nested state, but other formats can also be used.

[0049] As shown in Figure 6, the item definition section P21 of prompt P2 has multiple items as item names ("fieldName"), including department name, visit date, activity time (from), activity time (to), requirements, progress status, interviewer, and detailed content. Here, the multiple items in prompt P2 may also include an item (first item) that asks for content to be selected from a list of multiple input candidates. In Figure 6, the first item corresponds to the item where "list" is written in the "dataType" column for each item, and specifically, the item names "Requirements" and "Progress Status" correspond to the first item, respectively.

[0050] The item name "Requirements" is listed in the "dataList" column as comma-separated options, for example, "Visit, Visit (Leaving Business Card), Telephone / Email, Internal Office Work, Visit to Company, Head Office Meeting / Training, Other, Accompanying / Being Present, Document Receipt, Telesales". The content of the item name "Requirements" is selected from these listed options. The selection criteria are described in the "hint" column, which defines the content of the output, for example, "Ignore the schedule. If you are unsure, select Visit."

[0051] The item name "Progress Status" has comma-separated candidates listed in the "dataList" column, for example, "Initial, Meeting, In Progress, Interrupted, Contract Closed". The content of the item name "Progress Status" is selected from these listed options. The selection criteria are written in the "hint" column, which defines the content of the output, for example, "How much progress has the project made as a result of the visit? Do not consider it a contract unless there is content similar to a contract. A preliminary agreement is not a contract." In this way, for items whose content is selectively determined from a limited set of terms, the limited terms can be listed in advance in prompt P2 to provide multiple input candidates. This allows for the content of each item in the output text data P3, which will be described later, to be appropriate.

[0052] Furthermore, in prompt P2, multiple items may have an item (second item) that requests text content indicating the date and time. Here, "date and time" includes cases where it is only the day or only the time. In Figure 6, items with "date" or "time" written in the dataType column of each item correspond to the second item, and specifically, the item names "Visit Date," "Activity Time (from)," and "Activity Time (to)" correspond to the second item, respectively. The item definition section P21 may specify that the date and time should be output in the format specified in the "dataFormat" column (for example, "yyyy / MM / dd" and "HH:mm," etc.).

[0053] The criteria for the date and time to be output are specified in the "hint" field, which defines the content of the output. For example, it can be specified as "Date of visit; if not specified, today's date. Today is February 16, 2024." By outputting the date and time data as text in a predetermined format, it can be used as the content to be displayed on the screen in that format. This makes it easier and more accurate to create documents such as reports, meeting minutes, and other records.

[0054] Furthermore, multiple items in prompt P2 may be classified into one of three categories: the first item, the second item, or a third item that does not fall under either the first or second item. In Figure 6, items with "text" written in the "dataType" column may correspond to the third item. The item definition section P21 specifies that the output should be in text format by writing "text" in the "dataType" column. Specifically, these are the item names "Department Name," "Interviewee," and "Detailed Content." The criteria for the content to be written in text format are written in the "hint" column which defines the content of the output. For example, in the "Interviewee" column, it may be specified as "Please leave a space between the names and titles of those who attended the meeting during the visit. If there are multiple names, please put them in an array format."

[0055] Furthermore, the third item may be anything other than the content selected from the multiple listed input candidates or the text content indicating the date and time. In this way, the creator of prompt P2 can create prompt P2 in any of the first, second, or third way of writing. This allows the creator of prompt P2 to more easily create prompt P2 by determining which of the first to third items an item corresponds to.

[0056] Furthermore, in prompt P2, multiple items may include items that specify what to output if the specific content contained in the audio information does not exist. Specifically, in Figure 6, for example, the item named "Visit Date" specifies, "The date of the visit, or today's date if not specified. Today is February 16, 2024." In this case, if the audio information contains the date of the visit, the date of the visit is determined as the content of the "Visit Date" item. However, if the audio information does not contain the date of the visit, it can be set to today's date, that is, the date the audio information was acquired.

[0057] Furthermore, for example, in the item named "Requirements" in Figure 6, it is stipulated that "If you are unsure, please arrange a visit." In this case, if the voice information contains content corresponding to the "Requirements," that content will be determined as the content of the "Requirements." However, if the voice information does not contain content corresponding to the Requirements, "A visit" can be determined as the content of the "Requirements."

[0058] Normally, if there is no specific content to be entered into an item based on the voice information, the item's input content can be left blank, allowing the person in charge to manually enter it. However, even when specific content is missing, by pre-defining other input content, the system can output content corresponding to the item. This makes it easier to create documents such as reports, meeting minutes, and other records. Note that "when specific content is missing" may also include cases where the language model 5 was unable to recognize that specific content.

[0059] Furthermore, the specifications regarding the output content of the prompt P2 items may include the current date and time. Similar to the "date and time" mentioned above, "date and time" can include cases where only the day is entered or only the time is entered. For example, in Figure 6, "Today is February 16, 2024." in the item name "Visit Date" corresponds to the current date and time as the day only, and "The current time is 13:47." in the item name "Activity Time (from)" corresponds to the current date and time as the time only. This allows the output content to be date and time based on the time prompt P2 was created or sent, even when the output content includes the current date and time. Also, for example, even if time has passed between the time the date and time are entered in prompt P2 and the processing of language model 5 due to processing constraints, if the correct date and time are entered in prompt P2, output text data P3 containing the correct date and time can be output.

[0060] Thus, prompt P2 defines the content and format of the output text data P3, which consists of multiple items. Such prompt P2 is provided to the language model 5 along with the audio information and reflected in the output text data P3. Prompt P2 may also indicate hints for data generation, data length, and data format. Here, "format" and "data length and data format" can, for example, define the format of the item content, such as data size, whether it is fixed-length or variable-length, the (maximum or fixed) number of characters if it is a string, and the types of characters such as numbers and alphabets.

[0061] In this embodiment, a prompt P2 is generated based on the selected item definition portion P21, and this prompt P2 is provided to the language model 5 along with the voice information. Here, the server device 1 may also generate prompt P2 by adding the following preamble portion to the top of the item definition portion P21 of prompt P2 shown in Figure 6.

[0062] "You will identify specific keywords or phrases within the input string, select the appropriate information based on them, and transform it. This transformation includes extracting content corresponding to the specified fieldName and hint, and processing it based on the rules defined in length, dataType, dataFormat, and dataList. The results will be provided in JSON format. Please strictly adhere to hint, length, dataType, and dataFormat. Please also output the criteria for each item." The execution steps are as follows: 1. The input string will be analyzed to identify the content corresponding to fieldName or hint. Strict adherence to the hint is required. 2. For each fieldName, apply the appropriate format according to the dataType (text, date, time, list, etc.). For date and time, convert according to the specific dataFormat (e.g., "yyyy / MM / dd" or "HH:mm"). 3. For fieldName with dataList, select from the provided candidates and check if it matches the specified option. 4. Apply length-based restrictions to each field's Name as needed. 5. Construct the extracted and transformed information based on the above steps in JSON format.

[0063] In this embodiment, the item definition portion P21 and the preamble portion for prompt P2 are separated. However, the entire prompt P2 may be stored in the storage unit 14, and program P1 may select the entire prompt P2 in step S102 described above.

[0064] As described above, the server device 1 accepts input of the type of information to be registered, which is the type of information to be registered, based on the instructions of program P1, and can select one prompt P2 from several types of prompts P2 that have been stored in advance, based on this registration information type. It can also provide the selected prompt P2 to the language model 5 along with the voice information. Here, "selecting one prompt P2" includes selecting a part of the multiple types of prompts P2 as described above and combining it with the common part to form a single prompt P2. This makes it possible to create documents such as reports, minutes, and other records more easily and accurately.

[0065] Figure 7 shows an example of output text data P3 consisting of multiple items. The output text data P3 in Figure 7 is composed of a combination of the name and content of each item. As shown in this figure, the output text data P3 has a "value" item added to each item in the item definition section P21 of Figure 6, and the content of "value" is entered based on the content of the audio information. The format of the output text data P3 can use a format that allows for the storage of each item and the value of each item in combination.

[0066] For example, a format such as JSON, as shown in Figure 7, can be used, which allows for the storage of each item and its value combination in an enumerated or nested state. By specifying a predetermined format for reports, minutes, and other records in prompt P2, output text data P3 in the specified format can be generated. This makes the creation of reports, minutes, and other records easier and more accurate. The output text data P3 may be acquired and generated by the server device 1 via the generation server device 4 and network 3.

[0067] Figure 8 shows an example of the registration information type selection screen 100. The registration information type selection screen 100 is displayed on the display unit 26 of the terminal device 2. As shown in this figure, the registration information type selection screen 100 has an item display area 101 in which items for selecting the event (registration information type) to record are listed. The item display area 101 can include items such as visit memos, maintenance records, order data registration, provisional reservation applications, and inquiries. Here, "visit memos" is an item for recording visits when visiting a customer. "Maintenance records" is an item for recording maintenance performed on equipment, etc.

[0068] "Order Data Registration" is an item for recording the type and quantity of parts that need to be ordered when the need arises to order parts, etc. "Provisional Reservation Request" is an item for recording the equipment to be used, the date and time of use, etc., when the need arises to use the equipment. "Inquiry" is an item for recording when the need arises to make an inquiry. Users of terminal device 2 can select any of these items and notify server device 1.

[0069] Here, the selection operation may be based on voice input or by operating a touch panel or the like. In the example of the registration information type selection screen 100 in Figure 8, an example using voice input is shown, indicating that voice is being acquired, and a voice input stop button 116 is displayed to stop voice input by tapping the screen.

[0070] Figure 9 shows an example of the input screen 200. The input screen 200 is displayed on the display unit 26 of the terminal device 2. As shown in this figure, the input screen 200 displays a title 201 indicating that the registration information type is "Visit Memo". The input screen 200 has an input item display area 203. Here, the input item display area 203 can reflect and display the item names of prompt P2, name of department, date of visit, activity time (from), activity time (to), requirements, progress status, interviewer, and details, as input items.

[0071] Furthermore, the input screen 200 may display a voice input start icon 215 that accepts the command to start voice input. Here, the touch panel, which serves as the input unit 27, may be arranged to function on the screen of the display unit 26. This allows the user of the terminal device 2 to start voice input by tapping the location where the voice input start icon 215 is displayed. Note that the input unit 27 that accepts the command to start voice input is not limited to a touch panel, but may also be a mouse, keyboard, etc. If the user is already operating using voice input, a voice input stop button may be displayed.

[0072] In this way, the input screen 200 can display a screen showing multiple items to be entered, as well as a voice input start icon 215 that accepts the command to start voice input. This allows the user to visually understand that they can input multiple items by voice. Furthermore, it enables the creation of documents such as reports, meeting minutes, and other records to be done more easily and accurately.

[0073] Figure 10 shows an example of the converted text display screen 300. The converted text display screen 300 is displayed on the display unit 26 of the terminal device 2. The converted text display screen 300 has a text display area 301 that sequentially displays the text data for which text conversion has been completed using the currently input voice. It may also have a voice input stop button 316 that indicates that voice acquisition is in progress and allows the user to stop voice input by tapping the screen.

[0074] Here, stopping voice input may be done by voice input indicating that voice input should be stopped. In this case, the decision of whether to stop voice input may be made by voice recognition processing performed by terminal device 2 or server device 1 that receives voice information from terminal device 2. Note that when voice data is directly input to language model 5, the converted text display screen 300 does not need to be displayed.

[0075] Figure 11 shows an example of the output text data list screen 400. The output text data list screen 400 is displayed on the display unit 26 of the terminal device 2. As shown in this figure, the output text data list screen 400 has an audio information list area 401 that displays a list of audio information to be processed. The audio information list area 401 may display a list of audio information for which the analysis processing request in step S108 was made.

[0076] In this case, for example, the audio information can be displayed in order of newest requests for analysis processing or in order of newest audio information, and the date and time, registration information type, and status (whether or not it has been analyzed) can be displayed respectively. The audio information list area 401 may display a list of audio information from which output text data P3 has been acquired. By selecting analyzed audio information through operations such as tapping on the terminal device 2, the output result screen 500 can be displayed.

[0077] Figure 12 shows an example of the output result screen 500. The output result screen 500 is displayed on the display unit 26 of the terminal device 2. The input item display area 503 of the output result screen 500 reflects the contents of the output text data P3 generated by the server device 1. Specifically, the items in the input item display area 503, name of department, date of visit, activity time (from), activity time (to), requirements, progress status, interviewer, and details, each reflect the contents of the item names corresponding to the items in the output text data P3 shown in Figure 7.

[0078] As shown in Figure 12, the output result screen 500, like the input screen 200, has a title 501 and an input item display area 503. The output result screen 500 also has a voice input start icon 515, similar to the input screen 200. The voice input start icon 515 accepts the operation to start voice input, similar to the input screen 200. Furthermore, the output result screen 500 has a playback icon 517 that accepts the operation to start playback of the audio data.

[0079] The playback icon 517 accepts the command to start playback of audio data stored in the memory unit 14 when voice input is received. If a touch panel is used as the input unit 27, the user of the terminal device 2 can initiate the start of the corresponding function by tapping the display position of each icon. Note that the input unit 27 is not limited to a touch panel, but may also be a mouse, keyboard, etc.

[0080] Here, for example, if the content of the item name "Detailed Content" in the output text data P3 in Figure 7 has a large number of characters and cannot be displayed in the "Detailed Content" field of the input item display area 203 of the output result screen 500 in Figure 12, only a portion can be displayed to indicate that input has been entered. In this way, the output result screen 500 can display at least a portion of the text corresponding to each of the multiple items in the output text data P3 in the corresponding fields for each of the multiple items.

[0081] In this case, the server device 1 can receive an instruction to select one of the multiple items shown on the output result screen 500 and display all of the text corresponding to the selected item. This allows the user to understand whether or not there is text for an item on the output result screen 500, even if the number of characters of text to be displayed for an item is large, and to visually understand the operation required to display the entire text.

[0082] Figure 13 shows an example of an editing screen 600 that displays all the text corresponding to a single item. In Figure 13, the editing screen 600 displays all the content text for the item name "Detailed Content" from the output result screen 500 in Figure 12. As shown in Figure 13, the editing screen 600 has a content display area 610. The editing screen 600 also has an edit button 601, a save button 602, a playback icon 603, and a voice input start icon 604.

[0083] The edit button 601 is used to edit the content of an item displayed in the content display area 610. Tapping the edit button 601 makes the content of the content display area 610 editable. The save button 602 is used to save the edited content. The functions of the playback icon 603 and the voice input start icon 604 are the same as those of the playback icon 517 and the voice input start icon 515 on the output result screen 500 described above, so redundant explanations are omitted.

[0084] Furthermore, since the server device 1 saves the received audio as an audio file, by tapping the playback icon 603, etc., it can play the saved audio file based on the instructions of program P1, and at the same time accept an instruction to save the edited content of one of the multiple items, and save the edited content. Playing the audio file here includes the server device 1 causing the terminal device 2 to perform streaming playback.

[0085] This allows users to check the audio file even if they doubt whether the content of the output result screen 500, which is generated based on the output text data P3, is appropriate. They can also correct it if necessary. Furthermore, even if they forget the content of the voice input at a later date, they can play the audio file to check it.

[0086] The editing screen 600 can be accessed by tapping or other operations on the corresponding item displayed in the output result screen 500. The operation is not limited to touch panel operation; it may also be performed using a mouse or keyboard. Furthermore, the editing screen 600 may be displayed for the purpose of editing content even if content is already displayed in the input item display area 503 of the output result screen 500.

[0087] Furthermore, the output result screen 500 may, for example, accept instructions and display speech-to-text data obtained by converting the input speech into text. Specifically, an operation object such as an icon that accepts instructions to display speech-to-text data is placed on the output result screen 500, and the server device 1 that executes program P1 accepts instructions to display speech-to-text data by operating the operation object. As a result, it becomes possible to access the speech-to-text data that is the basis of the content displayed on the output result screen 500 from the output result screen 500, so even if there are blanks or unclear points in the content of the output result screen 500, the content of the speech-to-text data can be easily checked.

[0088] [Differentiation] Figure 14 shows an example of a system configuration without a generation server device 4. In Figure 1, the generation server device 4 is provided separately from the server device 1, but as shown in Figure 14, the server device 1 may directly input voice information to the language model 5 without a generation server device 4. Here, a prompt P2 may also be input along with the voice information.

[0089] [Embodiment 2] Figure 15 is a schematic diagram of the record creation system according to Embodiment 2. In Embodiment 1, the server device 1, which is the information processing device 1, executes the program P1, but in Embodiment 2, the terminal device 2, which is the information processing device 2, executes the program P1. As shown in this figure, the record creation system in Figure 15, compared to the record creation system in Figure 1, lacks the information processing device (server device) 1 and consists of the information processing device (terminal device) 2, the generation server device 4, the network 3, and the language model 5.

[0090] Figure 16 is a block diagram showing an example configuration of terminal device 2 according to Embodiment 2. The hardware configuration of terminal device 2 is the same as that shown in Figure 4, except that the storage unit 24 stores a program (or program product) P1 and prompt P2 executed by the control unit 21. Program P1 and prompt P2 have the same content as the program P1 and prompt P2 stored in the storage unit 14 of server device 1 in Embodiment 1, and can output the same output to the display unit 26 as in Embodiment 1, and output the same screen.

[0091] In the record creation system shown in Figures 15 and 16, the terminal device 2 can operate in the same procedure as the flowchart in Figure 5 by replacing the operation of the server device 1 based on program P1 with the operation of the terminal device 2 based on program P1. In this case, in Embodiment 2, communication via the network 3 between the server device 1 and the terminal device 2 as in Embodiment 1 does not occur, and processing is performed within the terminal device 2.

[0092] As a result, the terminal device 2, similar to the server device 1 in Embodiment 1, can receive audio related to an event after the event has ended based on the instructions of the program P1 stored in the storage unit 24, provide the audio information, which is audio-based information, to the language model 5, and generate output text data P3 consisting of multiple items including the date and circumstances. Therefore, similar to Embodiment 1, output text data P3 can be generated for multiple items included in a predetermined format such as a report, minutes, or other record, making it easier and more accurate to create documents such as records.

[0093] Furthermore, the terminal device 2 can provide the language model 5 with a prompt P2, along with voice information, that defines the content and format of the output text data P3 of multiple items that will be generated based on the instructions of the program P1. In addition, the prompt P2 may indicate hints for data generation, the length of the data, and the data format. For example, the terminal device 2 may have a prompt P2 in its storage unit 24 that includes the item definition portion P21 shown in Figure 6.

[0094] Furthermore, the terminal device 2 can accept input of the type of information to be registered, which is the type of information to be registered, based on the instructions of program P1 (Figure 5, step S102), select one prompt P2 from several types of prompts P2 that have been stored in advance based on the type of information to be registered (Figure 5, step S103), and provide the selected prompt P2 to the language model 5 along with the voice information (Figure 5, step S108).

[0095] Furthermore, in prompt P2, the multiple items may include, for example, a first item selected from a list of multiple input candidates, and a second item consisting of text indicating the date and time, as shown in the item definition portion P21 included in prompt P2 in Figure 6. In this case, the multiple items may be classified into one of the first item, the second item, or a third item that does not fall under either the first or second item.

[0096] Furthermore, in prompt P2, specifically, as shown in the item name "Visit Date" in Figure 6, multiple items may include items that specify what to output regarding specific content included in the audio information, as well as items that specify what to output when no such specific content exists.

[0097] Furthermore, the terminal device 2 can display an input screen 200 (see Figure 9) that shows multiple items to be entered based on the instructions of program P1, as well as a voice input start icon 215 (515, 604) that accepts the operation to start voice input.

[0098] The output result screen 500 may be displayed showing at least a portion of the text corresponding to each of the multiple items in the output text data P3 in the corresponding fields for each of the multiple items, and the system may accept an instruction to select one of the multiple items shown on the output result screen 500 and then display the entire text corresponding to the selected item (see Figures 12 and 13).

[0099] Furthermore, terminal device 2 can save the received audio to an audio file based on the instructions of program P1. In addition, while playing back the saved audio file, it can receive a command to save the edited content of one of several items and save the edited content (see Figure 13, playback icon 603, etc.).

[0100] Furthermore, the terminal device 2 can display an output result screen 500 that shows at least a portion of the text corresponding to each of the multiple items in the output text data P3 in the corresponding fields for each of the multiple items, based on the instructions of program P1. The output result screen 500 can also show a link to access the audio file (see Figure 12).

[0101] In Embodiment 2, the contents of the output result screen 500 can also be accessed from multiple terminal devices by saving the contents of the output result screen 500 in a way that makes it accessible to a server device or the like that is always connected to the network 3.

[0102] In the embodiments described above, the cases in which program P1 is executed on server device 1 and on terminal device 2 were shown respectively. However, program P1 may be stored in a distributed manner on server device 1 and terminal device 2, and server device 1 and terminal device 2 may cooperate to operate the entire program P1, thereby configuring a single information processing device with multiple devices.

[0103] According to the program P1, the information processing method related to program P1, and the information processing device 1 and / or 2 of the embodiments of this disclosure, reports, minutes, and other records can be created more easily and accurately. The program P1 may be referred to as a program product, software, or software product, which may be provided on a recording medium or provided in a form distributed via a communication network.

[0104] The embodiments described herein are illustrative in all respects and not restrictive. The scope of the invention is not limited to those described herein but is defined by the claims, and all modifications within the meaning and scope equivalent to the claims are intended.

[0105] The matters described in each embodiment can be combined with each other. Furthermore, the independent and dependent claims described in the claims can be combined with each other in any combination, regardless of the form of reference. In addition, although the claims do not use the form of a multi-claim (multi-multi-claim) which is a claim that references two or more claims, it is possible to use a combination in which each claim of the same category references all of its parent claims in the form of a multi-multi-claim. [Explanation of Symbols]

[0106] 1. Information processing device (server device) 11 Control Unit 12 Communications Department 13 Reading Unit 14 Storage section 19 bus 1a Portable storage medium 2. Information Processing Device (Terminal Device) 21 Control Unit 22 Communications Department 23 Reading section 24 Memory section 26 Display section 27 Input section 28 Mike 29 bus 2a Portable storage media 3 Network 4. Generation Server Device 41 Control Unit 42 Communications Department 43 Reading Unit 44 Storage section 46 Display section 49 bus 4a Portable storage media 5 Language Models P1 Program (Program Product) P2 Prompt P21 Item definition part P3 Output Text Data 100 Registration Information Type Selection Screen 101 Item display area 116 Voice input stop button 200 Input screen 201 Title 203 Input Item Display Area 215 Voice input start icon 300 Converted Text Display Screen 301 Text display area 316 Voice input stop button 400 Output Text Data List Screen 401 Audio Information List Area 500 Output Result Screen 501 Title 503 Input Item Display Area 515 Voice input start icon 517 Playback icon 600 Editing screen 601 Edit button 602 Save button 603 Playback icon 604 Voice input start icon 610 Content display area

Claims

1. After the event ends, we will accept audio recordings related to the aforementioned event. The aforementioned audio-based information is provided to the language model to generate output text data consisting of multiple items, including the date and situation. A program that instructs a computer to perform a process.

2. The language model is given prompts, along with the speech information, that define the content and format of the output text data for the multiple items that are generated. The program according to claim 1.

3. The prompt indicates hints for generating data, the length of the data, and the data format. The program according to claim 2.

4. The system accepts input for the type of information to be registered, which is the type of information to be registered. Based on the registration information type, select one of the multiple types of prompts that have been saved in advance, The selected prompt is provided to the language model along with the voice information. The program according to claim 2.

5. The multiple items of the prompt are, The first item asks for the selection from multiple listed input candidates, The second item requests text content indicating the date and time, and includes The program according to claim 2.

6. The aforementioned multiple items are classified into one of the following: the first item, the second item, or a third item that does not fall under either the first or second item. The program according to claim 5.

7. The multiple items of the prompt specify that a particular content included in the audio information should be output, and also include an item that specifies what to output if the particular content does not exist. The program according to claim 2.

8. The screen displays the multiple items to be entered, as well as a voice input start icon that accepts the operation to start voice input. The program according to claim 1.

9. The output result screen displays at least a portion of the text corresponding to each of the multiple items in the output text data, in the corresponding fields for each of the multiple items. The system accepts an instruction to select one of the multiple items shown on the output result screen. Display all of the text corresponding to the selected item. The program according to claim 1.

10. The received audio is saved as an audio file. While playing the saved audio file, the system accepts a request to save the edited content of one of the multiple items. Save the edited content. The program according to claim 1.

11. The output result screen displays at least a portion of the text corresponding to each of the multiple items in the output text data, in the corresponding fields for each of the multiple items. The output result screen shows a link that allows access to the audio file. The program according to claim 10.

12. After the event ends, we will accept audio recordings related to the aforementioned event. The aforementioned audio-based information is provided to the language model to generate output text data consisting of multiple items, including the date and situation. Information processing methods.

13. It includes a control unit, and the control unit is After the event ends, we will accept audio recordings related to the aforementioned event. The aforementioned audio-based information is provided to the language model to generate output text data consisting of multiple items, including the date and situation. Information processing device.

Images