System

Abstract

A system is provided.SOLUTION: A system comprising: speech recognition means; means for converting speech data into text data; means for analyzing the text data and generating a workflow diagram; means for displaying the generated workflow diagram; means for receiving additional information from a user; means for modifying the workflow diagram using the additional information; means for extracting a problem associated with the workflow and presenting a solution; means for converting the generated workflow diagram, problem, and solution into slides and animations; and means for providing the slides and animations to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Many local governments and private companies have not made much progress in in-house business process review and improvement activities (BPR), and many inefficient processes and paper-based work remain. Furthermore, because specialized knowledge is required, they are unavoidably reliant on external consultants, which is costly and time-consuming. Therefore, there is a demand for a system that allows users to easily create business flows using a natural conversational voice input format and intuitively grasp issues and solutions. [Means for solving the problem]

[0005] The present invention provides the following means to solve the above problems. First, a business process described orally by a user is received as voice data using a voice recognition means and converted into text data. Next, a means is included for analyzing the converted text data and automatically generating a business flow diagram. This business flow diagram is displayed on a terminal. Then, a means is provided for receiving additional information from the user as voice input, and a means is provided for correcting and supplementing the business flow diagram using the additional information. Furthermore, a means is included for extracting issues related to the business flow and presenting solutions. Finally, the problems are solved by building a system that includes a means for compiling the generated business flow diagram, issues, and solutions into slides, converting them into video format, and providing them to the user.

[0006] ---

[0007] "Speech recognition means" refers to technology or devices for converting voice data into text data.

[0008] "Text data" refers to a data format in which voice data is converted into character information.

[0009] A "business flow diagram" is a diagram that visually shows each step of a business process and depicts a series of steps.

[0010] "Display means" refers to the technology or device for visually presenting the generated workflow diagram and other information to the user.

[0011] "Additional information" refers to further explanation or detailed information provided by the user to correct or supplement the workflow diagram.

[0012] "Issue extraction means" refers to the technology or equipment used to analyze business flow diagrams and identify potential problems and bottlenecks.

[0013] "Solution presentation means" refers to technology or devices that present optimal solutions or improvement proposals for the extracted issues to users.

[0014] "Slide creation means" refers to the technology or device used to compile the generated business flow diagram, issues, and solutions in slide format.

[0015] "Video conversion means" refers to the technology or device used to convert created slides into video format and create a visual presentation.

[0016] "Means of delivery" refers to the technology or equipment used to deliver the final materials or information to the user. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] This invention is a system that supports business process redesign (BPR) in a conversational format using natural language, and mainly uses speech recognition technology, text analysis technology, and generative AI models to automatically generate and modify business flow diagrams, identify issues, and present solutions. This system operates mainly on a server, terminals, and users.

[0039] First, the user starts the voice input application and verbally explains the business flow using the device. For example, the user might explain a specific business process, such as, "In this department, we first receive the application form, check the contents, and then circulate it to our superiors." The device then records this and sends it to the server as voice data.

[0040] Next, the server uses a speech recognition means to convert the received voice data into text data. Based on this text data, an initial workflow diagram is generated. The generated workflow diagram is then displayed to the user via their terminal.

[0041] If the displayed workflow diagram has any branching points or missing information, the server detects this and notifies the user. The user then provides additional verbal explanation, which is then re-recorded on the device and sent to the server. The server then recognizes this speech again, converts it into text data, and corrects the workflow diagram.

[0042] Furthermore, the server uses a problem extraction method to extract related issues from the generated business flow diagram, identifying potential problems and areas for improvement. Based on this, it generates solutions and presents them to the user. These solutions include specific proposals, such as "We propose the introduction of an electronic approval system to speed up approvals."

[0043] Finally, the server compiles the final workflow diagram, problems, and solutions into slides, and then converts them into video format. These outputs are provided to the user via their device, allowing them to visually confirm the information.

[0044] Specific examples

[0045] 1. Voice Input and Recognition

[0046] The user verbally explains the business flow to the terminal, saying, "First, I will receive the application form, check the contents, and then pass it on to my supervisor."

[0047] The device records the audio and sends it to the server.

[0048] The server uses a voice recognition engine to convert the voice data into text data such as "First, I will receive the application form, check the contents, and then forward it to my supervisor."

[0049] 2. Generate a workflow diagram

[0050] The server analyzes the text data and generates a business flow diagram of "Accept → Confirm → Circulate."

[0051] The server presents the generated business flow diagram to the user.

[0052] 3. Modifying the Workflow Diagram

[0053] The user verbally explains the additional information, saying, "After circulation, once my superior approves, I will pass it on to the next department."

[0054] The device records additional audio and sends it to the server.

[0055] The server recognizes this voice and modifies the workflow diagram to "Accept → Confirm → Circulate → Approve → Send to next department."

[0056] 4. Presenting the problem and solution

[0057] The server extracts the problem that "approvals are prone to delays" and generates a solution that "proposes the introduction of an electronic approval system to speed up approvals."

[0058] The server converts the final business flow diagram, problems, and solutions into slides and videos and provides them to the user.

[0059] In this way, the present invention is a system that efficiently and intuitively supports a series of processes for implementing BPR in a conversational format using natural language.

[0060] The processing flow will be explained below.

[0061] The program processing will be specifically explained below by dividing it into steps.

[0062] ---

[0063] Step 1:

[0064] The user starts the voice input application on the device and verbally explains the business flow, for example, "In this department, we first receive the application form, check its contents, and then forward it to the superior."

[0065] Step 2:

[0066] The device records the user's voice and, when the recording is finished, sends the voice data to the server.

[0067] Step 3:

[0068] The server sends the received voice data to a voice recognition engine, which converts the voice data into text data, which is then stored in a database.

[0069] Step 4:

[0070] The server analyzes the stored text data and uses a contextual analysis engine to extract each step of the process.

[0071] Step 5:

[0072] The server automatically generates an initial workflow diagram based on the extracted business steps, and presents the generated workflow diagram to the user.

[0073] Step 6:

[0074] The server analyzes the business flow diagram to determine whether there are any branching points or missing information. If any branching points or missing information are detected, the server notifies the user.

[0075] Step 7:

[0076] The user verbally provides additional explanations about branching points and missing information through a voice input application, for example, "After circulation, once the supervisor approves, we will pass it on to the next department."

[0077] Step 8:

[0078] The device records additional audio and sends it back to the server.

[0079] Step 9:

[0080] The server recognizes the received additional voice data and converts it into text data.Then, it modifies the workflow diagram based on the received additional information.

[0081] Step 10:

[0082] The server uses an issue extraction engine to identify potential problems and bottlenecks based on the business flow diagram.

[0083] Step 11:

[0084] The server automatically generates solutions based on the extracted issues, making specific proposals such as "Propose the introduction of an electronic approval system to speed up approvals."

[0085] Step 12:

[0086] The server compiles the final business flow diagram, issues, and solutions into slides, which are then converted into video format.

[0087] Step 13:

[0088] The server transmits the generated slides and videos to the user's terminal.

[0089] Step 14:

[0090] The user reviews the provided slides and videos to identify areas for improvement and measures for business processes.

[0091] ---

[0092] These are the specific processing steps of the program. This process allows users to review and improve their business processes efficiently simply by explaining the workflow in natural language.

[0093] Example 1

[0094] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0095] In conventional business process redesign (BPR), understanding the current state of business processes, identifying issues, and proposing improvement measures are often done manually or on paper, requiring time and effort. Furthermore, when using only voice input, the voice data is not converted into text, and the creation and revision of business flow diagrams are not automated, making it difficult to efficiently redesign business processes. Furthermore, because the extraction of issues and the generation of solutions are also manual, they can be subjective and lack comprehensiveness.

[0096] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0097] In this invention, the server includes a speech recognition unit, a unit for converting speech data into text data, and a unit for analyzing the text data and generating a workflow diagram, which makes it possible to generate and modify workflows, extract problems, and present solutions simply by voice input.

[0098] "Speech recognition means" refers to technical devices or software for analyzing input voice data and converting it into text data.

[0099] "Means for converting voice data into text data" refers to technical devices or software that use speech recognition means to analyze voice data and convert it into human-readable text data.

[0100] "Means for analyzing text data and generating a workflow diagram" refers to technical devices or software that analyzes the content of text data and automatically generates a workflow diagram that visually shows business procedures.

[0101] "Means for presenting the generated workflow diagram" refers to technical devices or software that allow a user to visually confirm the generated workflow diagram.

[0102] "Means for receiving additional information from the user" means any technical device or software that accepts additional audio or text information provided by the user.

[0103] "Means for modifying a workflow diagram with additional information" refers to technical devices and software for updating and modifying an existing workflow diagram based on additional information provided by a user.

[0104] "Means for extracting issues related to business flow and proposing solutions" refers to technical devices and software for analyzing the generated business flow diagram, identifying potential problems, and generating and proposing solutions to those problems.

[0105] "Means for converting the generated workflow diagrams, problems, and solutions into a media format" refers to technical devices and software for converting the generated workflow diagrams, problems, and solutions into a visual format such as slides or videos.

[0106] "Means for providing users with data in media format" refers to technical devices and software that allow data converted into a media format to be displayed on a user's terminal or made available for download.

[0107] "Means for extracting issues and generating solutions using a technical model" refers to means for analyzing a business flow diagram using a specific technical model, extracting issues, and generating solutions.

[0108] "Means for proposing solutions to problems using generative AI models" refers to technical devices and software that use generative AI models to automatically generate specific solutions to problems and propose them to users.

[0109] This invention is a system that supports business process redesign (BPR) in a conversational format using natural language, and mainly uses speech recognition technology, text analysis technology, and generative AI models to automatically generate and modify business flow diagrams, identify issues, and present solutions. This system operates mainly on a server, terminals, and users.

[0110] First, the user launches a voice input application on their device. For example, a laptop or smartphone can be used. The user verbally explains the business flow into the device, for example, "First, receive the application form, check the contents, and then circulate it to your superior." The device then records this and sends it to the server as voice data. This process uses a high-performance microphone and a voice input application.

[0111] Next, the server uses a speech recognition engine (for example, Google Cloud Speech-to-Text) to convert the received voice data into text data. The speech recognition engine analyzes the voice data and converts it into an appropriate text format using a language model. An initial workflow diagram is generated based on this text data. The server uses a text analysis method (for example, a Python natural language processing library) to analyze the text data and generate an initial workflow diagram of "Accept → Confirm → Circulate."

[0112] The generated workflow diagram is displayed to the user via the terminal. For example, the workflow diagram is displayed graphically using a web browser or a dedicated viewer application. If the user determines that the displayed workflow diagram needs to be corrected, additional voice explanations are provided. For example, additional information is provided, such as "After circulation, once the supervisor approves, we will pass it on to the next department." The terminal records the additional voice and sends it back to the server.

[0113] The server performs voice recognition again and converts it into text data. Based on this, the workflow diagram is revised and updated to the following order: "Accept → Confirm → Circulate → Approve → Send to next department." Through this series of processes, an accurate workflow diagram is created that matches the user's intentions.

[0114] Furthermore, the server uses an issue extraction method to extract relevant issues from the generated business flow diagram and identifies potential problems and areas for improvement. For example, it automatically detects the issue of "approval delays." The server then uses a generative AI model (e.g., OpenAI GPT-4) to generate solutions such as "We propose the introduction of an electronic approval system to speed up approvals." An example of a prompt to be input to the generative AI model is "Please identify the problems with this business flow diagram and propose improvements."

[0115] Finally, the server converts the final business flow diagram, detected issues, and solutions into slides (e.g., PowerPoint) and video formats (e.g., MP4). These outputs are provided to users via their devices, allowing them to visually view the information. For example, they can view it on a laptop or smartphone screen. They can also be shared in a conference room using a projector.

[0116] This system allows users to redesign business processes efficiently and intuitively. By using natural language voice input, users can effortlessly generate and modify business flow diagrams, identify issues, and present solutions.

[0117] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0118] Step 1:

[0119] The user launches a voice input application on their device. At the user's workplace, they launch the application using a device such as a laptop or smartphone. The user then verbally explains the workflow. For example, they might say, "First, I will receive the application form, check its contents, and then circulate it to my superior." The device records this voice input through the microphone and saves it as voice data. This recorded voice data becomes the input.

[0120] Step 2:

[0121] The device sends the recorded voice data to the server over an internet connection. Specifically, the voice data is uploaded to the server using an HTTP POST request. The server receives the voice data and uses it as input for its speech recognition engine.

[0122] Step 3:

[0123] The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert the speech data into text data. Specifically, the server analyzes the speech data and applies a language model to generate text data. This converted text data is the output. The conversion from speech data to text data includes phonemic recognition and semantic analysis processes.

[0124] Step 4:

[0125] The server analyzes the text data and performs processing to generate an initial workflow diagram. Specifically, it uses natural language processing (NLP) technology to analyze the grammatical structure and meaning of the text. For example, from the expression "Receive application, confirm the contents, and circulate to superior," it generates a flow of "Accept → Confirm → Circulate." This workflow diagram is the output.

[0126] Step 5:

[0127] The server generates data to visualize the generated workflow diagram. For example, it generates image data in SVG or PNG format and sends it to the terminal. The terminal presents this workflow diagram to the user. The user visually checks the workflow diagram on the terminal screen.

[0128] Step 6:

[0129] If the user needs to provide additional explanations for the workflow diagram, they can use the voice input application again to provide the additional information verbally. For example, they can say, "After circulation, once the supervisor approves, we will pass it on to the next department." The device records this additional voice explanation and sends it back to the server. This additional voice data becomes the input.

[0130] Step 7:

[0131] The server converts the additional voice data into text data again using a speech recognition engine. The converted text data is analyzed and modifications are made to the existing workflow diagram. For example, the workflow diagram is updated to "Accept → Confirm → Circulate → Approve → Send to next department." This modified workflow diagram is output.

[0132] Step 8:

[0133] The server analyzes the revised workflow diagram and identifies potential problems using problem extraction methods. For example, it detects the problem of "approval is often delayed." This problem data is output.

[0134] Step 9:

[0135] The server uses a generative AI model (e.g., OpenAI GPT-4) to generate a solution to the problem. The prompt is "Please identify the problems in this workflow diagram and propose improvements," and the generated solution is output. For example, a proposed solution might be "We propose the introduction of an electronic approval system to speed up approvals."

[0136] Step 10:

[0137] The server converts the final business flow diagram, problems, and solutions into slide format (e.g., PowerPoint) and video format (e.g., MP4). These media format products are sent to the terminal. The terminal receives the sent data and provides it to the user. The user can visually check this information on the terminal screen.

[0138] In this way, the present invention is implemented as a system that efficiently supports the redesign of business processes by making full use of voice input and AI.

[0139] (Application example 1)

[0140] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0141] In production sites, optimizing production processes is difficult, and there are many inefficient manual tasks and wasteful operations. To solve these problems, there is a need for a support system that allows on-site operators and engineers to intuitively optimize production processes.

[0142] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0143] In this invention, the server includes a speech recognition unit, a unit for converting speech data into text data, a unit for analyzing the text data and generating a workflow diagram, a unit for displaying the generated workflow diagram, a unit for receiving additional information from a user, a unit for modifying the workflow diagram using the additional information, a unit for extracting issues related to the workflow and presenting solutions, a unit for converting the generated workflow diagram, issues, and solutions into slides and videos, a unit for providing the slides and videos to the user, a unit for automatically generating a production process flow diagram, extracting issues, and proposing solutions based on a voice-input explanation for optimizing the production process flow, and a unit for visualizing the results including the issues and solutions as industrial slides. This enables operators and engineers to optimize the production process through intuitive voice operations.

[0144] A "voice recognition means" is a device or software system for converting voice data into text data.

[0145] "Means for converting voice data into text data" refers to the technology or process for converting recorded voice data into text data as character information.

[0146] The "means for analyzing text data and generating a workflow diagram" refers to an algorithm or system for analyzing the converted text data and generating a workflow diagram that illustrates the flow and procedures of a business process.

[0147] The "means for displaying the generated workflow diagram" refers to a device or software for visually displaying to the user the workflow diagram generated based on the analysis.

[0148] A "means for receiving additional information from a user" is an interface or system for receiving supplemental information or correction requests provided by a user.

[0149] The "means for modifying a workflow diagram using additional information" refers to a system or algorithm for modifying an existing workflow diagram based on additional information received from a user.

[0150] The "means for extracting issues related to business flow and proposing solutions" is an analysis system that identifies potential problems and areas for improvement from the generated business flow diagram and proposes solutions based on that.

[0151] The "means for converting the generated business flow diagram, issues, and solutions into slides and videos" refers to a tool or system for outputting the business flow diagram, issues, and solutions as presentation slides and videos.

[0152] The "means for providing slides and videos to users" refers to a device or platform for providing the generated slides and videos to users in a viewable form.

[0153] "Means for automatically generating a production process flow chart based on voice-input explanations in order to optimize the production process flow, extracting issues, and proposing solutions" is a system that automatically generates a production process flow chart based on voice input from the user, identifies points that need improvement from the chart, and proposes solutions.

[0154] "Means for visualizing results including problems and solutions as industrial slides" refers to tools and systems for visually displaying extracted problems and proposed solutions as industrial presentation slides.

[0155] To realize a system based on this invention, multiple steps must be performed: voice input, voice recognition, text analysis, business flow diagram generation, problem extraction, solution presentation, and final slide generation. The hardware and software configuration and processing required to realize this system are described in detail below.

[0156] Hardware Configuration

[0157] 1. Smartphone: This device is used by the user for voice input.

[0158] 2. Server: A central computer that processes and analyzes audio data.

[0159] Software Configuration

[0160] 1. Speech recognition software: For example, use the SpeechRecognition library or Google's speech recognition API.

[0161] 2. Text analysis and generative AI models: Use OpenAI's API.

[0162] 3. Presentation generation tool: Uses the Python-pptx library.

[0163] Processing Description

[0164] 1. Voice Input and Recognition

[0165] The user verbally explains the production process using the smartphone's microphone, for example, "First, we receive the raw materials, then we conduct a quality inspection, then we start assembly. Finally, we inspect the products and prepare them for shipping."

[0166] The terminal records this voice and sends it to the server.

[0167] 2. Voice Recognition

[0168] The server converts the voice data into text data using a speech recognition engine (for example, Google's speech recognition API). For example, the server obtains text data such as "First, we receive the raw materials, then we conduct a quality inspection, then we start the assembly process. Finally, we inspect the products and prepare them for shipping."

[0169] 3. Creating a production process flow

[0170] The server uses OpenAI's generative AI model to analyze the received text data and generate a production process flow diagram. The prompt is "Generate a production process flow diagram based on the following sentence: First, receive the raw materials, then perform quality inspection, then start assembly, and finally inspect and prepare for shipment."

[0171] The AI ​​model outputs a flow diagram of "Acceptance → Quality inspection → Assembly → Inspection → Preparation for shipment."

[0172] 4. Modifying the flow diagram

[0173] If the generated flow diagram contains any deficiencies or errors, the server notifies the user. The user can then provide additional explanations by voice and send them to the server via their terminal. For example, the user can provide additional information such as, "Some parts need to be re-inspected after assembly."

[0174] The server then performs speech recognition on the speech and converts it into text. The additional information is reflected in the flow diagram, generating a revised diagram.

[0175] 5. Identifying issues and presenting solutions

[0176] The server then uses the AI ​​model again to extract issues from the generated production process flow diagram, such as delays in assembly work.

[0177] Then, based on that, they present solutions, such as proposing "prevent delays by promoting automation of assembly work."

[0178] 6. Slide and video generation

[0179] The server uses Python-pptx to compile the generated flow diagrams, issues, and solutions into slides, which can then be output as videos.

[0180] Finally, the product is provided to the user through a terminal.

[0181] Specific examples

[0182] For example, if a user describes, "First, we receive the raw materials, then we perform quality inspection, then we perform assembly work, and finally we inspect the products and prepare them for shipping," the system will operate as follows:

[0183] 1. Speech recognition of user's explanation

[0184] 2. Convert to text data

[0185] 3. Generate a production process flow diagram: "Receipt → Quality inspection → Assembly → Inspection → Shipping preparation"

[0186] 4. Receive additional information from the user and modify the flow diagram as needed.

[0187] 5. Identify issues and present solutions

[0188] 6. Organize the final results in slide format and provide them to the user

[0189] This system allows for efficient and intuitive optimization of production processes simply by using voice input via a smartphone.

[0190] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0191] Step 1:

[0192] The user verbally describes the production process using the smartphone's microphone. For example, "First, we receive the raw materials, then we conduct quality inspections, then we perform assembly work, and finally we inspect the products and prepare them for shipment." The input is voice data.

[0193] Step 2:

[0194] The terminal records the voice input by the user and sends it to the server. The input is voice data, and the output is a data file sent to the server.

[0195] Step 3:

[0196] The server uses a speech recognition engine (for example, Google's speech recognition API) to convert the received voice data into text data. Specifically, the speech recognition model analyzes the voice signal and generates a corresponding text string. The input is the voice data, and the output is the text data: "First, receive the raw materials, then perform quality inspection, then perform assembly work, and finally inspect and prepare for shipment."

[0197] Step 4:

[0198] The server uses a generative AI model (for example, OpenAI's GPT-3) to analyze the obtained text data and generate a production process flow diagram. The prompt is input to the model in the form of "Generate a production process flow diagram based on the following text: First, receive the raw materials, then perform quality inspection, then perform assembly work. Finally, perform inspection and prepare for shipment." The generative AI model analyzes the text and generates a flow diagram of "Receipt → Quality inspection → Assembly → Inspection → Prepare for shipment." The input is text data, and the output is a production process flow diagram.

[0199] Step 5:

[0200] The server displays the generated flow diagram on the terminal, allowing the user to visually confirm it. The input is the production process flow diagram, and the output is the flow diagram displayed on the terminal.

[0201] Step 6:

[0202] If there are any omissions or errors in the flow diagram, the user can provide additional explanations by voice input, for example, "Some parts need to be re-inspected after assembly." The input is voice data.

[0203] Step 7:

[0204] The device records additional audio data and sends it back to the server. The input is the audio data and the output is the data file sent to the server.

[0205] Step 8:

[0206] The server again uses the speech recognition engine to convert the additional voice data into text data. Specifically, the speech recognition model analyzes the voice signal and generates text such as, "Some parts need to be re-inspected after assembly." The input is voice data, and the output is text data.

[0207] Step 9:

[0208] The server modifies the workflow diagram based on the additional text data. For example, it modifies it to the format "Acceptance → Quality Inspection → Assembly → Re-inspection → Inspection → Shipping Preparation." The input is text data, and the output is the modified production process flow diagram.

[0209] Step 10:

[0210] The server extracts issues from the revised flow diagram and proposes solutions using a generative AI model. For example, a delay in assembly work is extracted as an issue, and "automation of assembly work" is proposed as a solution. The input is the revised production process flow diagram, and the output is the issue and solution.

[0211] Step 11:

[0212] The server uses Python-pptx to compile the generated production process flow diagrams, problems, and solutions into slides, which can then be presented to users in a visually easy-to-understand format. The input is text data containing problems and solutions, and the output is presentation slides.

[0213] Step 12:

[0214] Finally, the terminal provides the generated slides to the user, who can visually review them and take action to improve the production process if necessary. The input is the presentation slides, and the output is the visual information provided to the user.

[0215] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0216] This invention is a system that supports business process redesign (BPR) in a conversational format using natural language, and by combining speech recognition technology, text analysis technology, a generative AI model, and an emotion engine, it generates and modifies business flow diagrams, extracts issues, presents solutions, and makes adjustments based on the user's emotions. This system operates mainly on a server, terminals, and users.

[0217] First, the user starts the voice input application and verbally explains the business flow using the device. For example, they might say, "In this department, we first receive the application form, check the contents, and then circulate it to our superiors." The device records this and sends it to the server as voice data.

[0218] Next, the server converts the received voice data into text data using a speech recognition device. Based on this converted text data, an initial workflow diagram is generated. The generated workflow diagram is then displayed on the user's device.

[0219] The server also uses an emotion engine to recognize emotions from the user's voice. For example, if the user is nervous or excited during the explanation, the server can detect the user's emotional state and adjust the generation or modification of the workflow diagram accordingly.

[0220] If the displayed workflow diagram contains any branching points or missing information, the server detects this and notifies the user. The user then provides additional verbal explanation, which is recorded on the device and sent to the server. The server then recognizes the speech again, converts it into text data, and modifies the workflow diagram.

[0221] The server can also use an emotion engine to continue the dialogue depending on the user's emotional state: for example, if the user is feeling stressed, it can provide feedback to promote relaxation.

[0222] The server also uses the problem extraction engine to extract related issues from the generated business flow diagram. It then generates solutions based on these and presents them to the user. For example, specific proposals such as "We propose the introduction of an electronic approval system to speed up approvals."

[0223] Finally, the server creates slides containing the business flow diagram, issues, and solutions, and converts them into video format. The generated slides and videos are then provided to the user via their device. The user can visually check this information and identify areas for improvement and countermeasures for the business process.

[0224] Specific examples

[0225] 1. Voice Input and Recognition

[0226] The user verbally explains the business flow to the terminal, saying, "First, receive the application form, check the contents, and then forward it to your supervisor." The terminal records the voice and sends it to the server. The server uses a voice recognition engine to convert the voice data into text data that says, "First, receive the application form, check the contents, and then forward it to your supervisor."

[0227] 2. Generate a workflow diagram

[0228] The server analyzes the text data and generates a workflow diagram of "Accept → Confirm → Circulate." It then uses an emotion engine to analyze the user's emotional state and makes adjustments according to their emotions. The server then presents the generated workflow diagram to the user.

[0229] 3. Modifying the Workflow Diagram

[0230] The user verbally explains the additional information, saying, "After circulation, once my superior approves, I will pass it on to the next department." The device records the additional voice and sends it to the server. The server recognizes this and modifies the workflow diagram to "Accept → Confirm → Circulate → Approve → Pass on to the next department." The emotion engine also analyzes the user's emotional state and adjusts the feedback when providing additional explanations.

[0231] 4. Presenting the problem and solution

[0232] The server extracts the issue of "approval delays" and generates a solution: "Propose the introduction of an electronic approval system to speed up approvals." The server converts the final business flow diagram, issue, and solution into slides and videos and provides them to the user. The user reviews the provided slides and videos and identifies areas for improvement in the business process and measures to be taken.

[0233] In this way, the present invention provides a system that efficiently and intuitively supports a series of processes for implementing BPR in a conversational manner using natural language, including adjustments based on emotions.

[0234] The processing flow will be explained below.

[0235] The program processing will be specifically explained below by dividing it into steps.

[0236] ---

[0237] Step 1:

[0238] The user starts the voice input application on the device and verbally explains the business flow, for example, "In this department, we first receive the application form, check its contents, and then forward it to the superior."

[0239] Step 2:

[0240] The device records the user's voice and, when the recording is finished, sends the voice data to the server.

[0241] Step 3:

[0242] The server sends the received voice data to a voice recognition engine, which converts the voice data into text data, which is then stored in a database.

[0243] Step 4:

[0244] The server analyzes the stored text data and uses a contextual analysis engine to extract each step of the process.

[0245] Step 5:

[0246] The server automatically generates an initial workflow diagram based on the extracted business steps, and the generated workflow diagram is displayed on the user's device.

[0247] Step 6:

[0248] The server analyzes the business flow diagram to determine whether there are any branching points or missing information. If any branching points or missing information are detected, the server notifies the user.

[0249] Step 7:

[0250] The user verbally provides additional explanations about branching points and missing information through a voice input application, for example, "After circulation, once the supervisor approves, we will pass it on to the next department."

[0251] Step 8:

[0252] The device records additional audio and sends it back to the server.

[0253] Step 9:

[0254] The server recognizes the received additional voice data and converts it into text data.Then, it modifies the workflow diagram based on the received additional information.

[0255] Step 10:

[0256] The server uses an emotion engine to recognize emotions from the user's voice. For example, if the user is nervous or excited, the server detects that emotional state.

[0257] Step 11:

[0258] The server adjusts the generation and modification of the workflow diagram based on the extracted emotional state. For example, if the user is nervous, the server will adjust the feedback to be gentler.

[0259] Step 12:

[0260] The server uses an issue extraction engine to identify potential problems and bottlenecks based on the business flow diagram.

[0261] Step 13:

[0262] The server automatically generates solutions based on the extracted issues, making specific proposals such as "Propose the introduction of an electronic approval system to speed up approvals."

[0263] Step 14:

[0264] The server compiles the final business flow diagram, issues, and solutions into slides, which are then converted into video format.

[0265] Step 15:

[0266] The server transmits the generated slides and videos to the user's terminal.

[0267] Step 16:

[0268] The user reviews the provided slides and videos to identify areas for improvement and measures for business processes.

[0269] ---

[0270] These are the specific processing steps of the program. This process allows users to review and improve their business processes efficiently simply by explaining the workflow in natural language. The emotion engine provides responses and feedback based on the user's emotions, supporting more effective BPR.

[0271] Example 2

[0272] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0273] In conventional business process redesign (BPR) systems, when generating a workflow diagram from voice input, the process proceeds without taking into account the user's emotional state, which can lead to stress and misunderstandings. Another issue is the inefficiency of the problem extraction and solution presentation processes. Furthermore, there is a lack of a way to present the generated information visually in an easy-to-understand manner.

[0274] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a voice recognition means, a means for converting voice data into text data, and a means for recognizing the user's emotions and adjusting the generation and modification of a workflow diagram based on the emotions. This enables flexible responses according to the user's emotional state, and enables efficient generation and modification of a workflow diagram while reducing stress and misunderstandings.

[0275] "Speech recognition means" refers to a device or program for converting voice data into text data.

[0276] "Means for converting voice data into text data" refers to technology that analyzes recorded voice data and converts its contents into text information.

[0277] "Means for analyzing text data and generating business flow diagrams" refers to technology for extracting business procedures and flows from text data and generating a flowchart that visually represents them.

[0278] The "means for displaying the generated workflow diagram" refers to a device or program for displaying the generated workflow diagram on a terminal display or monitor.

[0279] The "means for receiving additional information from the user" is an interface that allows the user to provide further supplementary explanations or corrections to the workflow diagram verbally or by other means.

[0280] The "means for modifying a workflow diagram using additional information" is a technique for updating or modifying an existing workflow diagram based on additional information received from a user.

[0281] "Means for extracting issues related to business flow and proposing solutions" is a technology for finding problems in the generated business flow diagram and proposing specific solutions to them.

[0282] "Means for converting the generated business flow diagram, issues, and solutions into slides and videos" refers to technology for converting the generated business flow diagram, issues, and solutions into presentation slides or video format in order to present them in a visually easy-to-understand manner.

[0283] The "means for providing slides and videos to the user" refers to a technique for visually presenting the generated slides and videos to the user.

[0284] "Means for recognizing a user's emotions and adjusting the generation and modification of a workflow diagram based on those emotions" is a technology that analyzes the user's emotional state from their voice and actions, and adjusts the progress of the generation and modification of a workflow diagram based on those emotions.

[0285] This invention is a system that supports business process redesign (BPR) in a conversational format using natural language. This system combines speech recognition technology, text analysis technology, a generative AI model, and an emotion engine to generate and modify business flow diagrams, identify issues, present solutions, and make adjustments based on the user's emotions. This system operates primarily on a server, terminals, and users.

[0286] First, the user starts the voice input application and verbally explains the business flow using the device. For example, they might say, "In this department, we first receive the application form, check the contents, and then circulate it to our superiors." The device records this and sends it to the server as voice data.

[0287] Next, the server uses a speech recognition tool to convert the received voice data into text data. Based on this converted text data, an initial workflow diagram is generated. Using a generative AI model, the steps and flow of the business are extracted from the text data, and a flowchart that visually represents them is generated. The generated workflow diagram is then displayed on the user's device.

[0288] The server also uses an emotion engine to recognize emotions from the user's voice. For example, if the user is nervous or excited during the explanation, the server can detect that emotional state and adjust the generation or modification of the workflow diagram accordingly.

[0289] If the displayed workflow diagram contains any branching points or missing information, the server detects this and notifies the user. The user then provides additional verbal explanation, which is recorded on the device and sent to the server. The server then recognizes the speech again, converts it into text data, and modifies the workflow diagram.

[0290] The server also uses a problem extraction engine to extract related issues from the generated business flow diagram. Based on this, it generates solutions and presents them to the user. For example, specific proposals such as "We propose the introduction of an electronic approval system to speed up approvals" are included.

[0291] Finally, the server creates slides containing the business flow diagram, issues, and solutions, and converts them into video format. The generated slides and videos are then provided to the user via their device. The user can visually check this information and identify areas for improvement and countermeasures for the business process.

[0292] As a specific example, the following steps can be mentioned.

[0293] 1. Voice Input and Recognition

[0294] The user verbally explains the business flow to the terminal, saying, "First, receive the application form, check the contents, and then forward it to your supervisor." The terminal records the voice and sends it to the server. The server uses a voice recognition engine to convert the voice data into text data that says, "First, receive the application form, check the contents, and then forward it to your supervisor."

[0295] 2. Generate a workflow diagram

[0296] The server analyzes the text data and generates a workflow diagram of "Accept → Confirm → Circulate." It then uses an emotion engine to analyze the user's emotional state and makes adjustments according to their emotions. The server then presents the generated workflow diagram to the user.

[0297] 3. Modifying the Workflow Diagram

[0298] The user verbally explains the additional information, saying, "After circulation, once my superior approves, I will pass it on to the next department." The device records the additional voice and sends it to the server. The server recognizes this and modifies the workflow diagram to "Accept → Confirm → Circulate → Approve → Pass on to the next department." The emotion engine also analyzes the user's emotional state and adjusts the feedback when providing additional explanations.

[0299] 4. Presenting the problem and solution

[0300] The server extracts the issue of "approval delays" and generates a solution: "Propose the introduction of an electronic approval system to speed up approvals." The server converts the final business flow diagram, issue, and solution into slides and videos and provides them to the user. The user reviews the provided slides and videos and identifies areas for improvement in the business process and measures to be taken.

[0301] As described above, the present invention is a system that implements BPR in a conversational format using natural language, and provides efficient and intuitive support, including adjustments based on emotions.

[0302] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0303] Step 1:

[0304] The user starts a voice input application and verbally explains the business flow. For example, "In this department, we first receive the application form, check the contents, and then circulate it to our superiors." The input is voice data, and the output is a recorded voice file. The device records the user's voice and saves the voice data.

[0305] Step 2:

[0306] The device sends the recorded audio file to the server. The input is the recorded audio file, and the output is the audio data transferred to the server. The device converts the audio file to an appropriate format (e.g., WAV format) and sends it to the server via an HTTP request.

[0307] Step 3:

[0308] The server converts the received voice data into text data using a voice recognition engine (e.g., Google Speech-to-Text API). The input is voice data, and the output is text data. The server passes the voice file to the voice recognition engine and obtains the converted text data.

[0309] Step 4:

[0310] The server uses a generative AI model to extract the workflow from the converted text data and generate a workflow diagram. The input is text data, and the output is workflow diagram data. The server analyzes the text and extracts workflow steps such as "accept → confirm → circulate" based on the content.

[0311] Step 5:

[0312] The server uses an emotion engine to analyze the user's emotions. The input is voice data and text data, and the output is emotional state data. The server passes the voice data to the emotion engine for analysis and obtains results such as whether the user is nervous or excited.

[0313] Step 6:

[0314] The server displays the generated workflow diagram on the user's terminal. The input is the workflow diagram data, and the output is a visual display on the terminal. The server sends the workflow diagram to the user's terminal, and the terminal displays it on the screen.

[0315] Step 7:

[0316] The user verbally provides supplementary information to add explanations or make corrections to the business flow diagram. For example, the user might say, "After circulation, once my superior approves, I will pass it on to the next department." The input is additional voice data, and the output is the recorded supplementary voice data.

[0317] Step 8:

[0318] The device re-records the additional audio data and sends it to the server. The input is the additional audio data and the output is the supplemental audio data transferred to the server. The device again converts the audio file into an appropriate format and sends it to the server.

[0319] Step 9:

[0320] The server recognizes the additional voice data and converts it into text data. The input is the additional voice data and the output is the additional text data. The server uses a voice recognition engine to convert the additional explanation into text data.

[0321] Step 10:

[0322] The server modifies the workflow diagram based on the added text data. The input is the additional text data and the existing workflow diagram, and the output is the modified workflow diagram. Based on the additional explanation, the server updates the workflow diagram to "Accept → Confirm → Circulate → Approve → Send to next department."

[0323] Step 11:

[0324] The server uses the problem extraction engine to extract issues from the generated business flow diagram and presents solutions. The input is the modified business flow diagram, and the output is data on the issues and solutions. For example, it extracts the issue of "approval delays" and generates a solution such as "proposing the introduction of an electronic approval system."

[0325] Step 12:

[0326] The server creates slides based on the business flow diagram, issues, and solutions, and converts them into presentation slides and video formats. The input is the business flow diagram, issues, and solution data, and the output is the generated slides and videos. The server uses a slide creation tool to generate visually easy-to-understand slides and videos.

[0327] Step 13:

[0328] The server provides the generated slides and videos to the user's terminal. The input is slide and video data, and the output is a visual display on the user's terminal. The server sends the slides and videos to the user's terminal, and the user visually checks them.

[0329] The above are the processing steps of the program of this system, and summarize the specific operations, inputs, and outputs performed at each step.

[0330] (Application example 2)

[0331] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0332] Traditional business process redesign (BPR) is often highly manual, inefficient, and places a heavy burden on employees, especially those working in the field. It is also difficult to consider employees' emotional states, which can negatively impact stress and work efficiency. This invention aims to solve these problems and provide a system for redesigning business processes more efficiently and intuitively.

[0333] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0334] In this invention, the server includes a speech recognition unit, a means for converting speech data into text data, a means for analyzing the text data and generating a workflow diagram, a means for displaying the generated workflow diagram, a means for receiving additional information from a user, a means for modifying the workflow diagram using the additional information, a means for extracting issues related to the workflow and presenting solutions, a means for converting the generated workflow diagram, issues, and solutions into slides and videos, a means for providing the slides and videos to the user, an emotion recognition unit for analyzing the user's emotions from speech and providing feedback based on the emotional state, and a means for recording the user's voice when the user gives instructions on a business process and generating and modifying a workflow diagram. This allows on-site employees to intuitively generate and modify a workflow diagram simply by verbally explaining the business process, and provides optimal feedback according to the employee's emotional state, thereby improving work efficiency and reducing stress.

[0335] "Speech recognition means" is a technology that acquires and analyzes what a user says orally as digital voice data.

[0336] The "means for converting voice data into text data" refers to a technology for analyzing acquired digital voice data and converting it into corresponding text data.

[0337] The "means for analyzing text data and generating a business flow diagram" is a technology that automatically generates a flow diagram that visualizes the flow of business operations based on converted text data.

[0338] The "means for displaying the generated workflow diagram" is a technique for visually presenting the generated workflow diagram on the user's terminal.

[0339] The "means for receiving additional information from the user" refers to a technique for obtaining supplementary or corrective information provided by the user.

[0340] The "means for modifying a workflow diagram using additional information" is a technique for updating an existing workflow diagram based on additional information received from a user.

[0341] "Means for extracting issues related to business flow and proposing solutions" is a technology that finds problems in the generated business flow diagram and proposes solutions to those problems.

[0342] "Means for converting the generated business flow diagram, issues, and solutions into slides and videos" refers to a technology for converting business flow diagrams, extracted issues, and their solutions into visual slide and video formats.

[0343] The "means for providing slides and videos to users" refers to a technique for displaying or distributing the generated slides and videos to users.

[0344] "Emotion recognition means" is a technology that analyzes emotions from the user's voice and identifies their emotional state.

[0345] "Means for recording the voice of a user giving instructions on a business process and generating or modifying a business flow diagram" refers to a technology that records the user's verbal explanation of a business process and creates or modifies a business flow diagram based on that data.

[0346] This invention is a system that supports business process redesign (BPR) in a conversational format using natural language. By combining speech recognition technology, text analysis technology, a generative AI model, and an emotion engine, the system generates and modifies business flow diagrams, extracts issues, presents solutions, and makes adjustments based on the user's emotions. This system is mainly operated by a server, terminals, and users.

[0347] System configuration

[0348] 1. Voice recognition: The user verbally explains the business flow and the voice is captured as digital data. This is done using a smartphone with a built-in microphone.

[0349] 2. A method for converting voice data to text data: Analyze the voice data and convert it to the corresponding text data. This conversion is performed using the Python speech_recognition library.

[0350] 3. A method for analyzing text data and generating workflow diagrams: A workflow diagram is automatically generated based on the converted text data. A text generation model in Hugging Face's Transformers library is used.

[0351] 4. A means for displaying the generated workflow diagram: The generated workflow diagram is visually displayed on the user's device, using the smartphone screen.

[0352] 5. Means of receiving additional information from the user: A function for obtaining supplementary or corrective information provided by the user.

[0353] 6. A method for modifying a workflow diagram using additional information: Update an existing workflow diagram based on additional information from the user. This also uses speech recognition and text analysis technologies.

[0354] 7. A means to extract issues related to business processes and propose solutions: Identify problems in the generated business process diagram and propose solutions to those problems. This function uses a generative AI model.

[0355] 8. A means of converting generated workflow diagrams, issues, and solutions into slides and videos: A function to convert workflow diagrams and extracted issues and solutions into visual slide and video formats.

[0356] 9. Means of providing slides and videos to users: The ability to display or distribute the generated slides and videos to users.

[0357] 10. Emotion Recognition: Analyzes emotions from the user's voice and identifies their emotional state. This analysis is done using TextBlob and a customized emotion recognition engine, EmotionRecognizer.

[0358] 11. When giving instructions on a business process, a means for recording the voice and generating and modifying a business flow diagram: A user's verbal explanation of a business process is recorded, and a business flow diagram is created and modified based on that data.

[0359] Specific examples

[0360] 1. Voice Input and Recognition

[0361] The user verbally explains the workflow to the terminal, saying, "In this store, first we place products on the shelves, then we check the inventory and take our turn at the cash register." The terminal records the voice and sends it to the server, which uses a speech recognition engine to convert the voice data into text data.

[0362] 2. Generate a workflow diagram

[0363] The server analyzes the text data and generates a workflow diagram of "Shelf placement → Inventory check → Register shift." It then uses an emotion engine to analyze the user's emotional state and makes adjustments according to their emotions. The server then presents the generated workflow diagram to the user.

[0364] 3. Modifying the Workflow Diagram

[0365] The user verbally explains additional information, saying, "After the cashier changes, we will close the register." The device records the additional voice and sends it to the server. The server recognizes this and modifies the workflow diagram to "Shelf arrangement → Inventory check → Cashier change → Close the register." The emotion engine also analyzes the user's emotional state and adjusts the feedback when providing additional explanations.

[0366] 4. Presenting the problem and solution

[0367] The server extracts the issue that "cash register closing work is often delayed" and generates a solution that "proposes the introduction of an automatic cash register closing system." The server converts the final business flow diagram, issue, and solution into slides and videos and provides them to the user. The user reviews the provided slides and videos and understands the areas for improvement and countermeasures for the business process.

[0368] Prompt Sentence Examples

[0369] Current work flow: Place products on shelves, check inventory, and shift to the cash register.

[0370] Please give me some suggestions for improving this workflow. Please tell me specifically what to do, including the solution.

[0371] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0372] Step 1:

[0373] The user verbally explains the workflow, saying, "In this store, first we place products on the shelves, then we check the inventory and take turns at the cash register." The terminal records the voice and saves it as voice data. The input is the user's voice, and the output is voice data.

[0374] Step 2:

[0375] The server converts the voice data received from the terminal into text data using a voice recognition method. Specifically, it uses the Python speech_recognition library to analyze the voice data and generate Japanese text data. The input is voice data and the output is text data.

[0376] Step 3:

[0377] The server analyzes the text data and generates a workflow diagram. Using the text generation model in Hugging Face's Transformers library, a workflow diagram of "Shelf placement → Stock check → Register shift" is automatically generated from the text data. The input is text data, and the output is a workflow diagram.

[0378] Step 4:

[0379] The generated workflow diagram is sent to the user's terminal and displayed. The terminal visually presents the workflow diagram to the user. The input is the workflow diagram, and the output is the workflow diagram displayed on the terminal screen.

[0380] Step 5:

[0381] The user verbally explains changes to the workflow and additional information, such as "After the cashier change, we will perform the cashier closing work." The terminal records the additional voice and sends it to the server. The input is the user's voice, and the output is voice data.

[0382] Step 6:

[0383] The server converts the additional voice data into text data again using a voice recognition means, with the input being voice data and the output being text data.

[0384] Step 7:

[0385] The server reflects the additional text data in the original workflow diagram and modifies the workflow diagram to "Shelf placement → Stock check → Register shift → Cashier closing work." The input is the new text data and the existing workflow diagram, and the output is the modified workflow diagram.

[0386] Step 8:

[0387] The server extracts issues from the workflow diagram and proposes solutions using a generative AI model. For example, it extracts the issue "cash register closing work is often delayed" and proposes "the introduction of an automatic cash register closing system." The input is the workflow diagram, and the output is the issue and solution.

[0388] Step 9:

[0389] The generated workflow diagram, issues, and solutions are converted into slides and videos and sent to the user's device. The device visually presents the slides and videos to the user. The input is the workflow diagram, issues, and solutions, and the output is the slides and videos.

[0390] Step 10:

[0391] The server uses emotion recognition to analyze the emotions in the user's voice and provides feedback based on the user's emotional state. For example, if the user is feeling stressed, it generates feedback to encourage relaxation. The input is the user's voice and the output is feedback.

[0392] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0393] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0394] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0395] [Second embodiment]

[0396] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0397] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0398] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0399] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0400] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0401] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0402] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0403] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0404] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0405] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0406] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0407] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0408] This invention is a system that supports business process redesign (BPR) in a conversational format using natural language, and mainly uses speech recognition technology, text analysis technology, and generative AI models to automatically generate and modify business flow diagrams, identify issues, and present solutions. This system operates mainly on a server, terminals, and users.

[0409] First, the user starts the voice input application and verbally explains the business flow using the device. For example, the user might explain a specific business process, such as, "In this department, we first receive the application form, check the contents, and then circulate it to our superiors." The device then records this and sends it to the server as voice data.

[0410] Next, the server uses a speech recognition means to convert the received voice data into text data. Based on this text data, an initial workflow diagram is generated. The generated workflow diagram is then displayed to the user via their terminal.

[0411] If the displayed workflow diagram has any branching points or missing information, the server detects this and notifies the user. The user then provides additional verbal explanation, which is then re-recorded on the device and sent to the server. The server then recognizes this speech again, converts it into text data, and corrects the workflow diagram.

[0412] Furthermore, the server uses a problem extraction method to extract related issues from the generated business flow diagram, identifying potential problems and areas for improvement. Based on this, it generates solutions and presents them to the user. These solutions include specific proposals, such as "We propose the introduction of an electronic approval system to speed up approvals."

[0413] Finally, the server compiles the final workflow diagram, problems, and solutions into slides, and then converts them into video format. These outputs are provided to the user via their device, allowing them to visually confirm the information.

[0414] Specific examples

[0415] 1. Voice Input and Recognition

[0416] The user verbally explains the business flow to the terminal, saying, "First, I will receive the application form, check the contents, and then pass it on to my supervisor."

[0417] The device records the audio and sends it to the server.

[0418] The server uses a voice recognition engine to convert the voice data into text data such as "First, I will receive the application form, check the contents, and then forward it to my supervisor."

[0419] 2. Generate a workflow diagram

[0420] The server analyzes the text data and generates a business flow diagram of "Accept → Confirm → Circulate."

[0421] The server presents the generated business flow diagram to the user.

[0422] 3. Modifying the Workflow Diagram

[0423] The user verbally explains the additional information, saying, "After circulation, once my superior approves, I will pass it on to the next department."

[0424] The device records additional audio and sends it to the server.

[0425] The server recognizes this voice and modifies the workflow diagram to "Accept → Confirm → Circulate → Approve → Send to next department."

[0426] 4. Presenting the problem and solution

[0427] The server extracts the problem that "approvals are prone to delays" and generates a solution that "proposes the introduction of an electronic approval system to speed up approvals."

[0428] The server converts the final business flow diagram, problems, and solutions into slides and videos and provides them to the user.

[0429] In this way, the present invention is a system that efficiently and intuitively supports a series of processes for implementing BPR in a conversational format using natural language.

[0430] The processing flow will be explained below.

[0431] The program processing will be specifically explained below by dividing it into steps.

[0432] ---

[0433] Step 1:

[0434] The user starts the voice input application on the device and verbally explains the business flow, for example, "In this department, we first receive the application form, check its contents, and then forward it to the superior."

[0435] Step 2:

[0436] The device records the user's voice and, when the recording is finished, sends the voice data to the server.

[0437] Step 3:

[0438] The server sends the received voice data to a voice recognition engine, which converts the voice data into text data, which is then stored in a database.

[0439] Step 4:

[0440] The server analyzes the stored text data and uses a contextual analysis engine to extract each step of the process.

[0441] Step 5:

[0442] The server automatically generates an initial workflow diagram based on the extracted business steps, and presents the generated workflow diagram to the user.

[0443] Step 6:

[0444] The server analyzes the business flow diagram to determine whether there are any branching points or missing information. If any branching points or missing information are detected, the server notifies the user.

[0445] Step 7:

[0446] The user verbally provides additional explanations about branching points and missing information through a voice input application, for example, "After circulation, once the supervisor approves, we will pass it on to the next department."

[0447] Step 8:

[0448] The device records additional audio and sends it back to the server.

[0449] Step 9:

[0450] The server recognizes the received additional voice data and converts it into text data.Then, it modifies the workflow diagram based on the received additional information.

[0451] Step 10:

[0452] The server uses an issue extraction engine to identify potential problems and bottlenecks based on the business flow diagram.

[0453] Step 11:

[0454] The server automatically generates solutions based on the extracted issues, making specific proposals such as "Propose the introduction of an electronic approval system to speed up approvals."

[0455] Step 12:

[0456] The server compiles the final business flow diagram, issues, and solutions into slides, which are then converted into video format.

[0457] Step 13:

[0458] The server transmits the generated slides and videos to the user's terminal.

[0459] Step 14:

[0460] The user reviews the provided slides and videos to identify areas for improvement and measures for business processes.

[0461] ---

[0462] These are the specific processing steps of the program. This process allows users to review and improve their business processes efficiently simply by explaining the workflow in natural language.

[0463] Example 1

[0464] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0465] In conventional business process redesign (BPR), understanding the current state of business processes, identifying issues, and proposing improvement measures are often done manually or on paper, requiring time and effort. Furthermore, when using only voice input, the voice data is not converted into text, and the creation and revision of business flow diagrams are not automated, making it difficult to efficiently redesign business processes. Furthermore, because the extraction of issues and the generation of solutions are also manual, they can be subjective and lack comprehensiveness.

[0466] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0467] In this invention, the server includes a speech recognition unit, a unit for converting speech data into text data, and a unit for analyzing the text data and generating a workflow diagram, which makes it possible to generate and modify workflows, extract problems, and present solutions simply by voice input.

[0468] "Speech recognition means" refers to technical devices or software for analyzing input voice data and converting it into text data.

[0469] "Means for converting voice data into text data" refers to technical devices or software that use speech recognition means to analyze voice data and convert it into human-readable text data.

[0470] "Means for analyzing text data and generating a workflow diagram" refers to technical devices or software that analyzes the content of text data and automatically generates a workflow diagram that visually shows business procedures.

[0471] "Means for presenting the generated workflow diagram" refers to technical devices or software that allow a user to visually confirm the generated workflow diagram.

[0472] "Means for receiving additional information from the user" means any technical device or software that accepts additional audio or text information provided by the user.

[0473] "Means for modifying a workflow diagram with additional information" refers to technical devices and software for updating and modifying an existing workflow diagram based on additional information provided by a user.

[0474] "Means for extracting issues related to business flow and proposing solutions" refers to technical devices and software for analyzing the generated business flow diagram, identifying potential problems, and generating and proposing solutions to those problems.

[0475] "Means for converting the generated workflow diagrams, problems, and solutions into a media format" refers to technical devices and software for converting the generated workflow diagrams, problems, and solutions into a visual format such as slides or videos.

[0476] "Means for providing users with data in media format" refers to technical devices and software that allow data converted into a media format to be displayed on a user's terminal or made available for download.

[0477] "Means for extracting issues and generating solutions using a technical model" refers to means for analyzing a business flow diagram using a specific technical model, extracting issues, and generating solutions.

[0478] "Means for proposing solutions to problems using generative AI models" refers to technical devices and software that use generative AI models to automatically generate specific solutions to problems and propose them to users.

[0479] This invention is a system that supports business process redesign (BPR) in a conversational format using natural language, and mainly uses speech recognition technology, text analysis technology, and generative AI models to automatically generate and modify business flow diagrams, identify issues, and present solutions. This system operates mainly on a server, terminals, and users.

[0480] First, the user launches a voice input application on their device. For example, a laptop or smartphone can be used. The user verbally explains the business flow into the device, for example, "First, receive the application form, check the contents, and then circulate it to your superior." The device then records this and sends it to the server as voice data. This process uses a high-performance microphone and a voice input application.

[0481] Next, the server uses a speech recognition engine (for example, Google Cloud Speech-to-Text) to convert the received voice data into text data. The speech recognition engine analyzes the voice data and converts it into an appropriate text format using a language model. An initial workflow diagram is generated based on this text data. The server uses a text analysis method (for example, a Python natural language processing library) to analyze the text data and generate an initial workflow diagram of "Accept → Confirm → Circulate."

[0482] The generated workflow diagram is displayed to the user via the terminal. For example, the workflow diagram is displayed graphically using a web browser or a dedicated viewer application. If the user determines that the displayed workflow diagram needs to be corrected, additional voice explanations are provided. For example, additional information is provided, such as "After circulation, once the supervisor approves, we will pass it on to the next department." The terminal records the additional voice and sends it back to the server.

[0483] The server performs voice recognition again and converts it into text data. Based on this, the workflow diagram is revised and updated to the following order: "Accept → Confirm → Circulate → Approve → Send to next department." Through this series of processes, an accurate workflow diagram is created that matches the user's intentions.

[0484] Furthermore, the server uses an issue extraction method to extract relevant issues from the generated business flow diagram and identifies potential problems and areas for improvement. For example, it automatically detects the issue of "approval delays." The server then uses a generative AI model (e.g., OpenAI GPT-4) to generate solutions such as "We propose the introduction of an electronic approval system to speed up approvals." An example of a prompt to be input to the generative AI model is "Please identify the problems with this business flow diagram and propose improvements."

[0485] Finally, the server converts the final business flow diagram, detected issues, and solutions into slides (e.g., PowerPoint) and video formats (e.g., MP4). These outputs are provided to users via their devices, allowing them to visually view the information. For example, they can view it on a laptop or smartphone screen. They can also be shared in a conference room using a projector.

[0486] This system allows users to redesign business processes efficiently and intuitively. By using natural language voice input, users can effortlessly generate and modify business flow diagrams, identify issues, and present solutions.

[0487] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0488] Step 1:

[0489] The user launches a voice input application on their device. At the user's workplace, they launch the application using a device such as a laptop or smartphone. The user then verbally explains the workflow. For example, they might say, "First, I will receive the application form, check its contents, and then circulate it to my superior." The device records this voice input through the microphone and saves it as voice data. This recorded voice data becomes the input.

[0490] Step 2:

[0491] The device sends the recorded voice data to the server over an internet connection. Specifically, the voice data is uploaded to the server using an HTTP POST request. The server receives the voice data and uses it as input for its speech recognition engine.

[0492] Step 3:

[0493] The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert the speech data into text data. Specifically, the server analyzes the speech data and applies a language model to generate text data. This converted text data is the output. The conversion from speech data to text data includes phonemic recognition and semantic analysis processes.

[0494] Step 4:

[0495] The server analyzes the text data and performs processing to generate an initial workflow diagram. Specifically, it uses natural language processing (NLP) technology to analyze the grammatical structure and meaning of the text. For example, from the expression "Receive application, confirm the contents, and circulate to superior," it generates a flow of "Accept → Confirm → Circulate." This workflow diagram is the output.

[0496] Step 5:

[0497] The server generates data to visualize the generated workflow diagram. For example, it generates image data in SVG or PNG format and sends it to the terminal. The terminal presents this workflow diagram to the user. The user visually checks the workflow diagram on the terminal screen.

[0498] Step 6:

[0499] If the user needs to provide additional explanations for the workflow diagram, they can use the voice input application again to provide the additional information verbally. For example, they can say, "After circulation, once the supervisor approves, we will pass it on to the next department." The device records this additional voice explanation and sends it back to the server. This additional voice data becomes the input.

[0500] Step 7:

[0501] The server converts the additional voice data into text data again using a speech recognition engine. The converted text data is analyzed and modifications are made to the existing workflow diagram. For example, the workflow diagram is updated to "Accept → Confirm → Circulate → Approve → Send to next department." This modified workflow diagram is output.

[0502] Step 8:

[0503] The server analyzes the revised workflow diagram and identifies potential problems using problem extraction methods. For example, it detects the problem of "approval is often delayed." This problem data is output.

[0504] Step 9:

[0505] The server uses a generative AI model (e.g., OpenAI GPT-4) to generate a solution to the problem. The prompt is "Please identify the problems in this workflow diagram and propose improvements," and the generated solution is output. For example, a proposed solution might be "We propose the introduction of an electronic approval system to speed up approvals."

[0506] Step 10:

[0507] The server converts the final business flow diagram, problems, and solutions into slide format (e.g., PowerPoint) and video format (e.g., MP4). These media format products are sent to the terminal. The terminal receives the sent data and provides it to the user. The user can visually check this information on the terminal screen.

[0508] In this way, the present invention is implemented as a system that efficiently supports the redesign of business processes by making full use of voice input and AI.

[0509] (Application example 1)

[0510] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0511] In production sites, optimizing production processes is difficult, and there are many inefficient manual tasks and wasteful operations. To solve these problems, there is a need for a support system that allows on-site operators and engineers to intuitively optimize production processes.

[0512] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0513] In this invention, the server includes a speech recognition unit, a unit for converting speech data into text data, a unit for analyzing the text data and generating a workflow diagram, a unit for displaying the generated workflow diagram, a unit for receiving additional information from a user, a unit for modifying the workflow diagram using the additional information, a unit for extracting issues related to the workflow and presenting solutions, a unit for converting the generated workflow diagram, issues, and solutions into slides and videos, a unit for providing the slides and videos to the user, a unit for automatically generating a production process flow diagram, extracting issues, and proposing solutions based on a voice-input explanation for optimizing the production process flow, and a unit for visualizing the results including the issues and solutions as industrial slides. This enables operators and engineers to optimize the production process through intuitive voice operations.

[0514] A "voice recognition means" is a device or software system for converting voice data into text data.

[0515] "Means for converting voice data into text data" refers to the technology or process for converting recorded voice data into text data as character information.

[0516] The "means for analyzing text data and generating a workflow diagram" refers to an algorithm or system for analyzing the converted text data and generating a workflow diagram that illustrates the flow and procedures of a business process.

[0517] The "means for displaying the generated workflow diagram" refers to a device or software for visually displaying to the user the workflow diagram generated based on the analysis.

[0518] A "means for receiving additional information from a user" is an interface or system for receiving supplemental information or correction requests provided by a user.

[0519] The "means for modifying a workflow diagram using additional information" refers to a system or algorithm for modifying an existing workflow diagram based on additional information received from a user.

[0520] The "means for extracting issues related to business flow and proposing solutions" is an analysis system that identifies potential problems and areas for improvement from the generated business flow diagram and proposes solutions based on that.

[0521] The "means for converting the generated business flow diagram, issues, and solutions into slides and videos" refers to a tool or system for outputting the business flow diagram, issues, and solutions as presentation slides and videos.

[0522] The "means for providing slides and videos to users" refers to a device or platform for providing the generated slides and videos to users in a viewable form.

[0523] "Means for automatically generating a production process flow chart based on voice-input explanations in order to optimize the production process flow, extracting issues, and proposing solutions" is a system that automatically generates a production process flow chart based on voice input from the user, identifies points that need improvement from the chart, and proposes solutions.

[0524] "Means for visualizing results including problems and solutions as industrial slides" refers to tools and systems for visually displaying extracted problems and proposed solutions as industrial presentation slides.

[0525] To realize a system based on this invention, multiple steps must be performed: voice input, voice recognition, text analysis, business flow diagram generation, problem extraction, solution presentation, and final slide generation. The hardware and software configuration and processing required to realize this system are described in detail below.

[0526] Hardware Configuration

[0527] 1. Smartphone: This device is used by the user for voice input.

[0528] 2. Server: A central computer that processes and analyzes audio data.

[0529] Software Configuration

[0530] 1. Speech recognition software: For example, use the SpeechRecognition library or Google's speech recognition API.

[0531] 2. Text analysis and generative AI models: Use OpenAI's API.

[0532] 3. Presentation generation tool: Uses the Python-pptx library.

[0533] Processing Description

[0534] 1. Voice Input and Recognition

[0535] The user verbally explains the production process using the smartphone's microphone, for example, "First, we receive the raw materials, then we conduct a quality inspection, then we start assembly. Finally, we inspect the products and prepare them for shipping."

[0536] The terminal records this voice and sends it to the server.

[0537] 2. Voice Recognition

[0538] The server converts the voice data into text data using a speech recognition engine (for example, Google's speech recognition API). For example, the server obtains text data such as "First, we receive the raw materials, then we conduct a quality inspection, then we start the assembly process. Finally, we inspect the products and prepare them for shipping."

[0539] 3. Creating a production process flow

[0540] The server uses OpenAI's generative AI model to analyze the received text data and generate a production process flow diagram. The prompt is "Generate a production process flow diagram based on the following sentence: First, receive the raw materials, then perform quality inspection, then start assembly, and finally inspect and prepare for shipment."

[0541] The AI ​​model outputs a flow diagram of "Acceptance → Quality inspection → Assembly → Inspection → Preparation for shipment."

[0542] 4. Modifying the flow diagram

[0543] If the generated flow diagram contains any deficiencies or errors, the server notifies the user. The user can then provide additional explanations by voice and send them to the server via their terminal. For example, the user can provide additional information such as, "Some parts need to be re-inspected after assembly."

[0544] The server then performs speech recognition on the speech and converts it into text. The additional information is reflected in the flow diagram, generating a revised diagram.

[0545] 5. Identifying issues and presenting solutions

[0546] The server then uses the AI ​​model again to extract issues from the generated production process flow diagram, such as delays in assembly work.

[0547] Then, based on that, they present solutions, such as proposing "prevent delays by promoting automation of assembly work."

[0548] 6. Slide and video generation

[0549] The server uses Python-pptx to compile the generated flow diagrams, issues, and solutions into slides, which can then be output as videos.

[0550] Finally, the product is provided to the user through a terminal.

[0551] Specific examples

[0552] For example, if a user describes, "First, we receive the raw materials, then we perform quality inspection, then we perform assembly work, and finally we inspect the products and prepare them for shipping," the system will operate as follows:

[0553] 1. Speech recognition of user's explanation

[0554] 2. Convert to text data

[0555] 3. Generate a production process flow diagram: "Receipt → Quality inspection → Assembly → Inspection → Shipping preparation"

[0556] 4. Receive additional information from the user and modify the flow diagram as needed.

[0557] 5. Identify issues and present solutions

[0558] 6. Organize the final results in slide format and provide them to the user

[0559] This system allows for efficient and intuitive optimization of production processes simply by using voice input via a smartphone.

[0560] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0561] Step 1:

[0562] The user verbally describes the production process using the smartphone's microphone. For example, "First, we receive the raw materials, then we conduct quality inspections, then we perform assembly work, and finally we inspect the products and prepare them for shipment." The input is voice data.

[0563] Step 2:

[0564] The terminal records the voice input by the user and sends it to the server. The input is voice data, and the output is a data file sent to the server.

[0565] Step 3:

[0566] The server uses a speech recognition engine (for example, Google's speech recognition API) to convert the received voice data into text data. Specifically, the speech recognition model analyzes the voice signal and generates a corresponding text string. The input is the voice data, and the output is the text data: "First, receive the raw materials, then perform quality inspection, then perform assembly work, and finally inspect and prepare for shipment."

[0567] Step 4:

[0568] The server uses a generative AI model (for example, OpenAI's GPT-3) to analyze the obtained text data and generate a production process flow diagram. The prompt is input to the model in the form of "Generate a production process flow diagram based on the following text: First, receive the raw materials, then perform quality inspection, then perform assembly work. Finally, perform inspection and prepare for shipment." The generative AI model analyzes the text and generates a flow diagram of "Receipt → Quality inspection → Assembly → Inspection → Prepare for shipment." The input is text data, and the output is a production process flow diagram.

[0569] Step 5:

[0570] The server displays the generated flow diagram on the terminal, allowing the user to visually confirm it. The input is the production process flow diagram, and the output is the flow diagram displayed on the terminal.

[0571] Step 6:

[0572] If there are any omissions or errors in the flow diagram, the user can provide additional explanations by voice input, for example, "Some parts need to be re-inspected after assembly." The input is voice data.

[0573] Step 7:

[0574] The device records additional audio data and sends it back to the server. The input is the audio data and the output is the data file sent to the server.

[0575] Step 8:

[0576] The server again uses the speech recognition engine to convert the additional voice data into text data. Specifically, the speech recognition model analyzes the voice signal and generates text such as, "Some parts need to be re-inspected after assembly." The input is voice data, and the output is text data.

[0577] Step 9:

[0578] The server modifies the workflow diagram based on the additional text data. For example, it modifies it to the format "Acceptance → Quality Inspection → Assembly → Re-inspection → Inspection → Shipping Preparation." The input is text data, and the output is the modified production process flow diagram.

[0579] Step 10:

[0580] The server extracts issues from the revised flow diagram and proposes solutions using a generative AI model. For example, a delay in assembly work is extracted as an issue, and "automation of assembly work" is proposed as a solution. The input is the revised production process flow diagram, and the output is the issue and solution.

[0581] Step 11:

[0582] The server uses Python-pptx to compile the generated production process flow diagrams, problems, and solutions into slides, which can then be presented to users in a visually easy-to-understand format. The input is text data containing problems and solutions, and the output is presentation slides.

[0583] Step 12:

[0584] Finally, the terminal provides the generated slides to the user, who can visually review them and take action to improve the production process if necessary. The input is the presentation slides, and the output is the visual information provided to the user.

[0585] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0586] This invention is a system that supports business process redesign (BPR) in a conversational format using natural language, and by combining speech recognition technology, text analysis technology, a generative AI model, and an emotion engine, it generates and modifies business flow diagrams, extracts issues, presents solutions, and makes adjustments based on the user's emotions. This system operates mainly on a server, terminals, and users.

[0587] First, the user starts the voice input application and verbally explains the business flow using the device. For example, they might say, "In this department, we first receive the application form, check the contents, and then circulate it to our superiors." The device records this and sends it to the server as voice data.

[0588] Next, the server converts the received voice data into text data using a speech recognition device. Based on this converted text data, an initial workflow diagram is generated. The generated workflow diagram is then displayed on the user's device.

[0589] The server also uses an emotion engine to recognize emotions from the user's voice. For example, if the user is nervous or excited during the explanation, the server can detect the user's emotional state and adjust the generation or modification of the workflow diagram accordingly.

[0590] If the displayed workflow diagram contains any branching points or missing information, the server detects this and notifies the user. The user then provides additional verbal explanation, which is recorded on the device and sent to the server. The server then recognizes the speech again, converts it into text data, and modifies the workflow diagram.

[0591] The server can also use an emotion engine to continue the dialogue depending on the user's emotional state: for example, if the user is feeling stressed, it can provide feedback to promote relaxation.

[0592] The server also uses the problem extraction engine to extract related issues from the generated business flow diagram. It then generates solutions based on these and presents them to the user. For example, specific proposals such as "We propose the introduction of an electronic approval system to speed up approvals."

[0593] Finally, the server creates slides containing the business flow diagram, issues, and solutions, and converts them into video format. The generated slides and videos are then provided to the user via their device. The user can visually check this information and identify areas for improvement and countermeasures for the business process.

[0594] Specific examples

[0595] 1. Voice Input and Recognition

[0596] The user verbally explains the business flow to the terminal, saying, "First, receive the application form, check the contents, and then forward it to your supervisor." The terminal records the voice and sends it to the server. The server uses a voice recognition engine to convert the voice data into text data that says, "First, receive the application form, check the contents, and then forward it to your supervisor."

[0597] 2. Generate a workflow diagram

[0598] The server analyzes the text data and generates a workflow diagram of "Accept → Confirm → Circulate." It then uses an emotion engine to analyze the user's emotional state and makes adjustments according to their emotions. The server then presents the generated workflow diagram to the user.

[0599] 3. Modifying the Workflow Diagram

[0600] The user verbally explains the additional information, saying, "After circulation, once my superior approves, I will pass it on to the next department." The device records the additional voice and sends it to the server. The server recognizes this and modifies the workflow diagram to "Accept → Confirm → Circulate → Approve → Pass on to the next department." The emotion engine also analyzes the user's emotional state and adjusts the feedback when providing additional explanations.

[0601] 4. Presenting the problem and solution

[0602] The server extracts the issue of "approval delays" and generates a solution: "Propose the introduction of an electronic approval system to speed up approvals." The server converts the final business flow diagram, issue, and solution into slides and videos and provides them to the user. The user reviews the provided slides and videos and identifies areas for improvement in the business process and measures to be taken.

[0603] In this way, the present invention provides a system that efficiently and intuitively supports a series of processes for implementing BPR in a conversational manner using natural language, including adjustments based on emotions.

[0604] The processing flow will be explained below.

[0605] The program processing will be specifically explained below by dividing it into steps.

[0606] ---

[0607] Step 1:

[0608] The user starts the voice input application on the device and verbally explains the business flow, for example, "In this department, we first receive the application form, check its contents, and then forward it to the superior."

[0609] Step 2:

[0610] The device records the user's voice and, when the recording is finished, sends the voice data to the server.

[0611] Step 3:

[0612] The server sends the received voice data to a voice recognition engine, which converts the voice data into text data, which is then stored in a database.

[0613] Step 4:

[0614] The server analyzes the stored text data and uses a contextual analysis engine to extract each step of the process.

[0615] Step 5:

[0616] The server automatically generates an initial workflow diagram based on the extracted business steps, and the generated workflow diagram is displayed on the user's device.

[0617] Step 6:

[0618] The server analyzes the business flow diagram to determine whether there are any branching points or missing information. If any branching points or missing information are detected, the server notifies the user.

[0619] Step 7:

[0620] The user verbally provides additional explanations about branching points and missing information through a voice input application, for example, "After circulation, once the supervisor approves, we will pass it on to the next department."

[0621] Step 8:

[0622] The device records additional audio and sends it back to the server.

[0623] Step 9:

[0624] The server recognizes the received additional voice data and converts it into text data.Then, it modifies the workflow diagram based on the received additional information.

[0625] Step 10:

[0626] The server uses an emotion engine to recognize emotions from the user's voice. For example, if the user is nervous or excited, the server detects that emotional state.

[0627] Step 11:

[0628] The server adjusts the generation and modification of the workflow diagram based on the extracted emotional state. For example, if the user is nervous, the server will adjust the feedback to be gentler.

[0629] Step 12:

[0630] The server uses an issue extraction engine to identify potential problems and bottlenecks based on the business flow diagram.

[0631] Step 13:

[0632] The server automatically generates solutions based on the extracted issues, making specific proposals such as "Propose the introduction of an electronic approval system to speed up approvals."

[0633] Step 14:

[0634] The server compiles the final business flow diagram, issues, and solutions into slides, which are then converted into video format.

[0635] Step 15:

[0636] The server transmits the generated slides and videos to the user's terminal.

[0637] Step 16:

[0638] The user reviews the provided slides and videos to identify areas for improvement and measures for business processes.

[0639] ---

[0640] These are the specific processing steps of the program. This process allows users to review and improve their business processes efficiently simply by explaining the workflow in natural language. The emotion engine provides responses and feedback based on the user's emotions, supporting more effective BPR.

[0641] Example 2

[0642] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0643] In conventional business process redesign (BPR) systems, when generating a workflow diagram from voice input, the process proceeds without taking into account the user's emotional state, which can lead to stress and misunderstandings. Another issue is the inefficiency of the problem extraction and solution presentation processes. Furthermore, there is a lack of a way to present the generated information visually in an easy-to-understand manner.

[0644] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a voice recognition means, a means for converting voice data into text data, and a means for recognizing the user's emotions and adjusting the generation and modification of a workflow diagram based on the emotions. This enables flexible responses according to the user's emotional state, and enables efficient generation and modification of a workflow diagram while reducing stress and misunderstandings.

[0645] "Speech recognition means" refers to a device or program for converting voice data into text data.

[0646] "Means for converting voice data into text data" refers to technology that analyzes recorded voice data and converts its contents into text information.

[0647] "Means for analyzing text data and generating business flow diagrams" refers to technology for extracting business procedures and flows from text data and generating a flowchart that visually represents them.

[0648] The "means for displaying the generated workflow diagram" refers to a device or program for displaying the generated workflow diagram on a terminal display or monitor.

[0649] The "means for receiving additional information from the user" is an interface that allows the user to provide further supplementary explanations or corrections to the workflow diagram verbally or by other means.

[0650] The "means for modifying a workflow diagram using additional information" is a technique for updating or modifying an existing workflow diagram based on additional information received from a user.

[0651] "Means for extracting issues related to business flow and proposing solutions" is a technology for finding problems in the generated business flow diagram and proposing specific solutions to them.

[0652] "Means for converting the generated business flow diagram, issues, and solutions into slides and videos" refers to technology for converting the generated business flow diagram, issues, and solutions into presentation slides or video format in order to present them in a visually easy-to-understand manner.

[0653] The "means for providing slides and videos to the user" refers to a technique for visually presenting the generated slides and videos to the user.

[0654] "Means for recognizing a user's emotions and adjusting the generation and modification of a workflow diagram based on those emotions" is a technology that analyzes the user's emotional state from their voice and actions, and adjusts the progress of the generation and modification of a workflow diagram based on those emotions.

[0655] This invention is a system that supports business process redesign (BPR) in a conversational format using natural language. This system combines speech recognition technology, text analysis technology, a generative AI model, and an emotion engine to generate and modify business flow diagrams, identify issues, present solutions, and make adjustments based on the user's emotions. This system operates primarily on a server, terminals, and users.

[0656] First, the user starts the voice input application and verbally explains the business flow using the device. For example, they might say, "In this department, we first receive the application form, check the contents, and then circulate it to our superiors." The device records this and sends it to the server as voice data.

[0657] Next, the server uses a speech recognition tool to convert the received voice data into text data. Based on this converted text data, an initial workflow diagram is generated. Using a generative AI model, the steps and flow of the business are extracted from the text data, and a flowchart that visually represents them is generated. The generated workflow diagram is then displayed on the user's device.

[0658] The server also uses an emotion engine to recognize emotions from the user's voice. For example, if the user is nervous or excited during the explanation, the server can detect that emotional state and adjust the generation or modification of the workflow diagram accordingly.

[0659] If the displayed workflow diagram contains any branching points or missing information, the server detects this and notifies the user. The user then provides additional verbal explanation, which is recorded on the device and sent to the server. The server then recognizes the speech again, converts it into text data, and modifies the workflow diagram.

[0660] The server also uses a problem extraction engine to extract related issues from the generated business flow diagram. Based on this, it generates solutions and presents them to the user. For example, specific proposals such as "We propose the introduction of an electronic approval system to speed up approvals" are included.

[0661] Finally, the server creates slides containing the business flow diagram, issues, and solutions, and converts them into video format. The generated slides and videos are then provided to the user via their device. The user can visually check this information and identify areas for improvement and countermeasures for the business process.

[0662] As a specific example, the following steps can be mentioned.

[0663] 1. Voice Input and Recognition

[0664] The user verbally explains the business flow to the terminal, saying, "First, receive the application form, check the contents, and then forward it to your supervisor." The terminal records the voice and sends it to the server. The server uses a voice recognition engine to convert the voice data into text data that says, "First, receive the application form, check the contents, and then forward it to your supervisor."

[0665] 2. Generate a workflow diagram

[0666] The server analyzes the text data and generates a workflow diagram of "Accept → Confirm → Circulate." It then uses an emotion engine to analyze the user's emotional state and makes adjustments according to their emotions. The server then presents the generated workflow diagram to the user.

[0667] 3. Modifying the Workflow Diagram

[0668] The user verbally explains the additional information, saying, "After circulation, once my superior approves, I will pass it on to the next department." The device records the additional voice and sends it to the server. The server recognizes this and modifies the workflow diagram to "Accept → Confirm → Circulate → Approve → Pass on to the next department." The emotion engine also analyzes the user's emotional state and adjusts the feedback when providing additional explanations.

[0669] 4. Presenting the problem and solution

[0670] The server extracts the issue of "approval delays" and generates a solution: "Propose the introduction of an electronic approval system to speed up approvals." The server converts the final business flow diagram, issue, and solution into slides and videos and provides them to the user. The user reviews the provided slides and videos and identifies areas for improvement in the business process and measures to be taken.

[0671] As described above, the present invention is a system that implements BPR in a conversational format using natural language, and provides efficient and intuitive support, including adjustments based on emotions.

[0672] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0673] Step 1:

[0674] The user starts a voice input application and verbally explains the business flow. For example, "In this department, we first receive the application form, check the contents, and then circulate it to our superiors." The input is voice data, and the output is a recorded voice file. The device records the user's voice and saves the voice data.

[0675] Step 2:

[0676] The device sends the recorded audio file to the server. The input is the recorded audio file, and the output is the audio data transferred to the server. The device converts the audio file to an appropriate format (e.g., WAV format) and sends it to the server via an HTTP request.

[0677] Step 3:

[0678] The server converts the received voice data into text data using a voice recognition engine (e.g., Google Speech-to-Text API). The input is voice data, and the output is text data. The server passes the voice file to the voice recognition engine and obtains the converted text data.

[0679] Step 4:

[0680] The server uses a generative AI model to extract the workflow from the converted text data and generate a workflow diagram. The input is text data, and the output is workflow diagram data. The server analyzes the text and extracts workflow steps such as "accept → confirm → circulate" based on the content.

[0681] Step 5:

[0682] The server uses an emotion engine to analyze the user's emotions. The input is voice data and text data, and the output is emotional state data. The server passes the voice data to the emotion engine for analysis and obtains results such as whether the user is nervous or excited.

[0683] Step 6:

[0684] The server displays the generated workflow diagram on the user's terminal. The input is the workflow diagram data, and the output is a visual display on the terminal. The server sends the workflow diagram to the user's terminal, and the terminal displays it on the screen.

[0685] Step 7:

[0686] The user verbally provides supplementary information to add explanations or make corrections to the business flow diagram. For example, the user might say, "After circulation, once my superior approves, I will pass it on to the next department." The input is additional voice data, and the output is the recorded supplementary voice data.

[0687] Step 8:

[0688] The device re-records the additional audio data and sends it to the server. The input is the additional audio data and the output is the supplemental audio data transferred to the server. The device again converts the audio file into an appropriate format and sends it to the server.

[0689] Step 9:

[0690] The server recognizes the additional voice data and converts it into text data. The input is the additional voice data and the output is the additional text data. The server uses a voice recognition engine to convert the additional explanation into text data.

[0691] Step 10:

[0692] The server modifies the workflow diagram based on the added text data. The input is the additional text data and the existing workflow diagram, and the output is the modified workflow diagram. Based on the additional explanation, the server updates the workflow diagram to "Accept → Confirm → Circulate → Approve → Send to next department."

[0693] Step 11:

[0694] The server uses the problem extraction engine to extract issues from the generated business flow diagram and presents solutions. The input is the modified business flow diagram, and the output is data on the issues and solutions. For example, it extracts the issue of "approval delays" and generates a solution such as "proposing the introduction of an electronic approval system."

[0695] Step 12:

[0696] The server creates slides based on the business flow diagram, issues, and solutions, and converts them into presentation slides and video formats. The input is the business flow diagram, issues, and solution data, and the output is the generated slides and videos. The server uses a slide creation tool to generate visually easy-to-understand slides and videos.

[0697] Step 13:

[0698] The server provides the generated slides and videos to the user's terminal. The input is slide and video data, and the output is a visual display on the user's terminal. The server sends the slides and videos to the user's terminal, and the user visually checks them.

[0699] The above are the processing steps of the program of this system, and summarize the specific operations, inputs, and outputs performed at each step.

[0700] (Application example 2)

[0701] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0702] Traditional business process redesign (BPR) is often highly manual, inefficient, and places a heavy burden on employees, especially those working in the field. It is also difficult to consider employees' emotional states, which can negatively impact stress and work efficiency. This invention aims to solve these problems and provide a system for redesigning business processes more efficiently and intuitively.

[0703] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0704] In this invention, the server includes a speech recognition unit, a means for converting speech data into text data, a means for analyzing the text data and generating a workflow diagram, a means for displaying the generated workflow diagram, a means for receiving additional information from a user, a means for modifying the workflow diagram using the additional information, a means for extracting issues related to the workflow and presenting solutions, a means for converting the generated workflow diagram, issues, and solutions into slides and videos, a means for providing the slides and videos to the user, an emotion recognition unit for analyzing the user's emotions from speech and providing feedback based on the emotional state, and a means for recording the user's voice when the user gives instructions on a business process and generating and modifying a workflow diagram. This allows on-site employees to intuitively generate and modify a workflow diagram simply by verbally explaining the business process, and provides optimal feedback according to the employee's emotional state, thereby improving work efficiency and reducing stress.

[0705] "Speech recognition means" is a technology that acquires and analyzes what a user says orally as digital voice data.

[0706] The "means for converting voice data into text data" refers to a technology for analyzing acquired digital voice data and converting it into corresponding text data.

[0707] The "means for analyzing text data and generating a business flow diagram" is a technology that automatically generates a flow diagram that visualizes the flow of business operations based on converted text data.

[0708] The "means for displaying the generated workflow diagram" is a technique for visually presenting the generated workflow diagram on the user's terminal.

[0709] The "means for receiving additional information from the user" refers to a technique for obtaining supplementary or corrective information provided by the user.

[0710] The "means for modifying a workflow diagram using additional information" is a technique for updating an existing workflow diagram based on additional information received from a user.

[0711] "Means for extracting issues related to business flow and proposing solutions" is a technology that finds problems in the generated business flow diagram and proposes solutions to those problems.

[0712] "Means for converting the generated business flow diagram, issues, and solutions into slides and videos" refers to a technology for converting business flow diagrams, extracted issues, and their solutions into visual slide and video formats.

[0713] The "means for providing slides and videos to users" refers to a technique for displaying or distributing the generated slides and videos to users.

[0714] "Emotion recognition means" is a technology that analyzes emotions from the user's voice and identifies their emotional state.

[0715] "Means for recording the voice of a user giving instructions on a business process and generating or modifying a business flow diagram" refers to a technology that records the user's verbal explanation of a business process and creates or modifies a business flow diagram based on that data.

[0716] This invention is a system that supports business process redesign (BPR) in a conversational format using natural language. By combining speech recognition technology, text analysis technology, a generative AI model, and an emotion engine, the system generates and modifies business flow diagrams, extracts issues, presents solutions, and makes adjustments based on the user's emotions. This system is mainly operated by a server, terminals, and users.

[0717] System configuration

[0718] 1. Voice recognition: The user verbally explains the business flow and the voice is captured as digital data. This is done using a smartphone with a built-in microphone.

[0719] 2. A method for converting voice data to text data: Analyze the voice data and convert it to the corresponding text data. This conversion is performed using the Python speech_recognition library.

[0720] 3. A method for analyzing text data and generating workflow diagrams: A workflow diagram is automatically generated based on the converted text data. A text generation model in Hugging Face's Transformers library is used.

[0721] 4. A means for displaying the generated workflow diagram: The generated workflow diagram is visually displayed on the user's device, using the smartphone screen.

[0722] 5. Means of receiving additional information from the user: A function for obtaining supplementary or corrective information provided by the user.

[0723] 6. A method for modifying a workflow diagram using additional information: Update an existing workflow diagram based on additional information from the user. This also uses speech recognition and text analysis technologies.

[0724] 7. A means to extract issues related to business processes and propose solutions: Identify problems in the generated business process diagram and propose solutions to those problems. This function uses a generative AI model.

[0725] 8. A means of converting generated workflow diagrams, issues, and solutions into slides and videos: A function to convert workflow diagrams and extracted issues and solutions into visual slide and video formats.

[0726] 9. Means of providing slides and videos to users: The ability to display or distribute the generated slides and videos to users.

[0727] 10. Emotion Recognition: Analyzes emotions from the user's voice and identifies their emotional state. This analysis is done using TextBlob and a customized emotion recognition engine, EmotionRecognizer.

[0728] 11. When giving instructions on a business process, a means for recording the voice and generating and modifying a business flow diagram: A user's verbal explanation of a business process is recorded, and a business flow diagram is created and modified based on that data.

[0729] Specific examples

[0730] 1. Voice Input and Recognition

[0731] The user verbally explains the workflow to the terminal, saying, "In this store, first we place products on the shelves, then we check the inventory and take our turn at the cash register." The terminal records the voice and sends it to the server, which uses a speech recognition engine to convert the voice data into text data.

[0732] 2. Generate a workflow diagram

[0733] The server analyzes the text data and generates a workflow diagram of "Shelf placement → Inventory check → Register shift." It then uses an emotion engine to analyze the user's emotional state and makes adjustments according to their emotions. The server then presents the generated workflow diagram to the user.

[0734] 3. Modifying the Workflow Diagram

[0735] The user verbally explains additional information, saying, "After the cashier changes, we will close the register." The device records the additional voice and sends it to the server. The server recognizes this and modifies the workflow diagram to "Shelf arrangement → Inventory check → Cashier change → Close the register." The emotion engine also analyzes the user's emotional state and adjusts the feedback when providing additional explanations.

[0736] 4. Presenting the problem and solution

[0737] The server extracts the issue that "cash register closing work is often delayed" and generates a solution that "proposes the introduction of an automatic cash register closing system." The server converts the final business flow diagram, issue, and solution into slides and videos and provides them to the user. The user reviews the provided slides and videos and understands the areas for improvement and countermeasures for the business process.

[0738] Prompt Sentence Examples

[0739] Current work flow: Place products on shelves, check inventory, and shift to the cash register.

[0740] Please give me some suggestions for improving this workflow. Please tell me specifically what to do, including the solution.

[0741] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0742] Step 1:

[0743] The user verbally explains the workflow, saying, "In this store, first we place products on the shelves, then we check the inventory and take turns at the cash register." The terminal records the voice and saves it as voice data. The input is the user's voice, and the output is voice data.

[0744] Step 2:

[0745] The server converts the voice data received from the terminal into text data using a voice recognition method. Specifically, it uses the Python speech_recognition library to analyze the voice data and generate Japanese text data. The input is voice data and the output is text data.

[0746] Step 3:

[0747] The server analyzes the text data and generates a workflow diagram. Using the text generation model in Hugging Face's Transformers library, a workflow diagram of "Shelf placement → Stock check → Register shift" is automatically generated from the text data. The input is text data, and the output is a workflow diagram.

[0748] Step 4:

[0749] The generated workflow diagram is sent to the user's terminal and displayed. The terminal visually presents the workflow diagram to the user. The input is the workflow diagram, and the output is the workflow diagram displayed on the terminal screen.

[0750] Step 5:

[0751] The user verbally explains changes to the workflow and additional information, such as "After the cashier change, we will perform the cashier closing work." The terminal records the additional voice and sends it to the server. The input is the user's voice, and the output is voice data.

[0752] Step 6:

[0753] The server converts the additional voice data into text data again using a voice recognition means, with the input being voice data and the output being text data.

[0754] Step 7:

[0755] The server reflects the additional text data in the original workflow diagram and modifies the workflow diagram to "Shelf placement → Stock check → Register shift → Cashier closing work." The input is the new text data and the existing workflow diagram, and the output is the modified workflow diagram.

[0756] Step 8:

[0757] The server extracts issues from the workflow diagram and proposes solutions using a generative AI model. For example, it extracts the issue "cash register closing work is often delayed" and proposes "the introduction of an automatic cash register closing system." The input is the workflow diagram, and the output is the issue and solution.

[0758] Step 9:

[0759] The generated workflow diagram, issues, and solutions are converted into slides and videos and sent to the user's device. The device visually presents the slides and videos to the user. The input is the workflow diagram, issues, and solutions, and the output is the slides and videos.

[0760] Step 10:

[0761] The server uses emotion recognition to analyze the emotions in the user's voice and provides feedback based on the user's emotional state. For example, if the user is feeling stressed, it generates feedback to encourage relaxation. The input is the user's voice and the output is feedback.

[0762] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0763] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0764] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0765] [Third embodiment]

[0766] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0767] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0768] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0769] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0770] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0771] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0772] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0773] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0774] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0775] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0776] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0777] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0778] This invention is a system that supports business process redesign (BPR) in a conversational format using natural language, and mainly uses speech recognition technology, text analysis technology, and generative AI models to automatically generate and modify business flow diagrams, identify issues, and present solutions. This system operates mainly on a server, terminals, and users.

[0779] First, the user starts the voice input application and verbally explains the business flow using the device. For example, the user might explain a specific business process, such as, "In this department, we first receive the application form, check the contents, and then circulate it to our superiors." The device then records this and sends it to the server as voice data.

[0780] Next, the server uses a speech recognition means to convert the received voice data into text data. Based on this text data, an initial workflow diagram is generated. The generated workflow diagram is then displayed to the user via their terminal.

[0781] If the displayed workflow diagram has any branching points or missing information, the server detects this and notifies the user. The user then provides additional verbal explanation, which is then re-recorded on the device and sent to the server. The server then recognizes this speech again, converts it into text data, and corrects the workflow diagram.

[0782] Furthermore, the server uses a problem extraction method to extract related issues from the generated business flow diagram, identifying potential problems and areas for improvement. Based on this, it generates solutions and presents them to the user. These solutions include specific proposals, such as "We propose the introduction of an electronic approval system to speed up approvals."

[0783] Finally, the server compiles the final workflow diagram, problems, and solutions into slides, and then converts them into video format. These outputs are provided to the user via their device, allowing them to visually confirm the information.

[0784] Specific examples

[0785] 1. Voice Input and Recognition

[0786] The user verbally explains the business flow to the terminal, saying, "First, I will receive the application form, check the contents, and then pass it on to my supervisor."

[0787] The device records the audio and sends it to the server.

[0788] The server uses a voice recognition engine to convert the voice data into text data such as "First, I will receive the application form, check the contents, and then forward it to my supervisor."

[0789] 2. Generate a workflow diagram

[0790] The server analyzes the text data and generates a business flow diagram of "Accept → Confirm → Circulate."

[0791] The server presents the generated business flow diagram to the user.

[0792] 3. Modifying the Workflow Diagram

[0793] The user verbally explains the additional information, saying, "After circulation, once my superior approves, I will pass it on to the next department."

[0794] The device records additional audio and sends it to the server.

[0795] The server recognizes this voice and modifies the workflow diagram to "Accept → Confirm → Circulate → Approve → Send to next department."

[0796] 4. Presenting the problem and solution

[0797] The server extracts the problem that "approvals are prone to delays" and generates a solution that "proposes the introduction of an electronic approval system to speed up approvals."

[0798] The server converts the final business flow diagram, problems, and solutions into slides and videos and provides them to the user.

[0799] In this way, the present invention is a system that efficiently and intuitively supports a series of processes for implementing BPR in a conversational format using natural language.

[0800] The processing flow will be explained below.

[0801] The program processing will be specifically explained below by dividing it into steps.

[0802] ---

[0803] Step 1:

[0804] The user starts the voice input application on the device and verbally explains the business flow, for example, "In this department, we first receive the application form, check its contents, and then forward it to the superior."

[0805] Step 2:

[0806] The device records the user's voice and, when the recording is finished, sends the voice data to the server.

[0807] Step 3:

[0808] The server sends the received voice data to a voice recognition engine, which converts the voice data into text data, which is then stored in a database.

[0809] Step 4:

[0810] The server analyzes the stored text data and uses a contextual analysis engine to extract each step of the process.

[0811] Step 5:

[0812] The server automatically generates an initial workflow diagram based on the extracted business steps, and presents the generated workflow diagram to the user.

[0813] Step 6:

[0814] The server analyzes the business flow diagram to determine whether there are any branching points or missing information. If any branching points or missing information are detected, the server notifies the user.

[0815] Step 7:

[0816] The user verbally provides additional explanations about branching points and missing information through a voice input application, for example, "After circulation, once the supervisor approves, we will pass it on to the next department."

[0817] Step 8:

[0818] The device records additional audio and sends it back to the server.

[0819] Step 9:

[0820] The server recognizes the received additional voice data and converts it into text data.Then, it modifies the workflow diagram based on the received additional information.

[0821] Step 10:

[0822] The server uses an issue extraction engine to identify potential problems and bottlenecks based on the business flow diagram.

[0823] Step 11:

[0824] The server automatically generates solutions based on the extracted issues, making specific proposals such as "Propose the introduction of an electronic approval system to speed up approvals."

[0825] Step 12:

[0826] The server compiles the final business flow diagram, issues, and solutions into slides, which are then converted into video format.

[0827] Step 13:

[0828] The server transmits the generated slides and videos to the user's terminal.

[0829] Step 14:

[0830] The user reviews the provided slides and videos to identify areas for improvement and measures for business processes.

[0831] ---

[0832] These are the specific processing steps of the program. This process allows users to review and improve their business processes efficiently simply by explaining the workflow in natural language.

[0833] Example 1

[0834] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0835] In conventional business process redesign (BPR), understanding the current state of business processes, identifying issues, and proposing improvement measures are often done manually or on paper, requiring time and effort. Furthermore, when using only voice input, the voice data is not converted into text, and the creation and revision of business flow diagrams are not automated, making it difficult to efficiently redesign business processes. Furthermore, because the extraction of issues and the generation of solutions are also manual, they can be subjective and lack comprehensiveness.

[0836] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0837] In this invention, the server includes a speech recognition unit, a unit for converting speech data into text data, and a unit for analyzing the text data and generating a workflow diagram, which makes it possible to generate and modify workflows, extract problems, and present solutions simply by voice input.

[0838] "Speech recognition means" refers to technical devices or software for analyzing input voice data and converting it into text data.

[0839] "Means for converting voice data into text data" refers to technical devices or software that use speech recognition means to analyze voice data and convert it into human-readable text data.

[0840] "Means for analyzing text data and generating a workflow diagram" refers to technical devices or software that analyzes the content of text data and automatically generates a workflow diagram that visually shows business procedures.

[0841] "Means for presenting the generated workflow diagram" refers to technical devices or software that allow a user to visually confirm the generated workflow diagram.

[0842] "Means for receiving additional information from the user" means any technical device or software that accepts additional audio or text information provided by the user.

[0843] "Means for modifying a workflow diagram with additional information" refers to technical devices and software for updating and modifying an existing workflow diagram based on additional information provided by a user.

[0844] "Means for extracting issues related to business flow and proposing solutions" refers to technical devices and software for analyzing the generated business flow diagram, identifying potential problems, and generating and proposing solutions to those problems.

[0845] "Means for converting the generated workflow diagrams, problems, and solutions into a media format" refers to technical devices and software for converting the generated workflow diagrams, problems, and solutions into a visual format such as slides or videos.

[0846] "Means for providing users with data in media format" refers to technical devices and software that allow data converted into a media format to be displayed on a user's terminal or made available for download.

[0847] "Means for extracting issues and generating solutions using a technical model" refers to means for analyzing a business flow diagram using a specific technical model, extracting issues, and generating solutions.

[0848] "Means for proposing solutions to problems using generative AI models" refers to technical devices and software that use generative AI models to automatically generate specific solutions to problems and propose them to users.

[0849] This invention is a system that supports business process redesign (BPR) in a conversational format using natural language, and mainly uses speech recognition technology, text analysis technology, and generative AI models to automatically generate and modify business flow diagrams, identify issues, and present solutions. This system operates mainly on a server, terminals, and users.

[0850] First, the user launches a voice input application on their device. For example, a laptop or smartphone can be used. The user verbally explains the business flow into the device, for example, "First, receive the application form, check the contents, and then circulate it to your superior." The device then records this and sends it to the server as voice data. This process uses a high-performance microphone and a voice input application.

[0851] Next, the server uses a speech recognition engine (for example, Google Cloud Speech-to-Text) to convert the received voice data into text data. The speech recognition engine analyzes the voice data and converts it into an appropriate text format using a language model. An initial workflow diagram is generated based on this text data. The server uses a text analysis method (for example, a Python natural language processing library) to analyze the text data and generate an initial workflow diagram of "Accept → Confirm → Circulate."

[0852] The generated workflow diagram is displayed to the user via the terminal. For example, the workflow diagram is displayed graphically using a web browser or a dedicated viewer application. If the user determines that the displayed workflow diagram needs to be corrected, additional voice explanations are provided. For example, additional information is provided, such as "After circulation, once the supervisor approves, we will pass it on to the next department." The terminal records the additional voice and sends it back to the server.

[0853] The server performs voice recognition again and converts it into text data. Based on this, the workflow diagram is revised and updated to the following order: "Accept → Confirm → Circulate → Approve → Send to next department." Through this series of processes, an accurate workflow diagram is created that matches the user's intentions.

[0854] Furthermore, the server uses an issue extraction method to extract relevant issues from the generated business flow diagram and identifies potential problems and areas for improvement. For example, it automatically detects the issue of "approval delays." The server then uses a generative AI model (e.g., OpenAI GPT-4) to generate solutions such as "We propose the introduction of an electronic approval system to speed up approvals." An example of a prompt to be input to the generative AI model is "Please identify the problems with this business flow diagram and propose improvements."

[0855] Finally, the server converts the final business flow diagram, detected issues, and solutions into slides (e.g., PowerPoint) and video formats (e.g., MP4). These outputs are provided to users via their devices, allowing them to visually view the information. For example, they can view it on a laptop or smartphone screen. They can also be shared in a conference room using a projector.

[0856] This system allows users to redesign business processes efficiently and intuitively. By using natural language voice input, users can effortlessly generate and modify business flow diagrams, identify issues, and present solutions.

[0857] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0858] Step 1:

[0859] The user launches a voice input application on their device. At the user's workplace, they launch the application using a device such as a laptop or smartphone. The user then verbally explains the workflow. For example, they might say, "First, I will receive the application form, check its contents, and then circulate it to my superior." The device records this voice input through the microphone and saves it as voice data. This recorded voice data becomes the input.

[0860] Step 2:

[0861] The device sends the recorded voice data to the server over an internet connection. Specifically, the voice data is uploaded to the server using an HTTP POST request. The server receives the voice data and uses it as input for its speech recognition engine.

[0862] Step 3:

[0863] The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert the speech data into text data. Specifically, the server analyzes the speech data and applies a language model to generate text data. This converted text data is the output. The conversion from speech data to text data includes phonemic recognition and semantic analysis processes.

[0864] Step 4:

[0865] The server analyzes the text data and performs processing to generate an initial workflow diagram. Specifically, it uses natural language processing (NLP) technology to analyze the grammatical structure and meaning of the text. For example, from the expression "Receive application, confirm the contents, and circulate to superior," it generates a flow of "Accept → Confirm → Circulate." This workflow diagram is the output.

[0866] Step 5:

[0867] The server generates data to visualize the generated workflow diagram. For example, it generates image data in SVG or PNG format and sends it to the terminal. The terminal presents this workflow diagram to the user. The user visually checks the workflow diagram on the terminal screen.

[0868] Step 6:

[0869] If the user needs to provide additional explanations for the workflow diagram, they can use the voice input application again to provide the additional information verbally. For example, they can say, "After circulation, once the supervisor approves, we will pass it on to the next department." The device records this additional voice explanation and sends it back to the server. This additional voice data becomes the input.

[0870] Step 7:

[0871] The server converts the additional voice data into text data again using a speech recognition engine. The converted text data is analyzed and modifications are made to the existing workflow diagram. For example, the workflow diagram is updated to "Accept → Confirm → Circulate → Approve → Send to next department." This modified workflow diagram is output.

[0872] Step 8:

[0873] The server analyzes the revised workflow diagram and identifies potential problems using problem extraction methods. For example, it detects the problem of "approval is often delayed." This problem data is output.

[0874] Step 9:

[0875] The server uses a generative AI model (e.g., OpenAI GPT-4) to generate a solution to the problem. The prompt is "Please identify the problems in this workflow diagram and propose improvements," and the generated solution is output. For example, a proposed solution might be "We propose the introduction of an electronic approval system to speed up approvals."

[0876] Step 10:

[0877] The server converts the final business flow diagram, problems, and solutions into slide format (e.g., PowerPoint) and video format (e.g., MP4). These media format products are sent to the terminal. The terminal receives the sent data and provides it to the user. The user can visually check this information on the terminal screen.

[0878] In this way, the present invention is implemented as a system that efficiently supports the redesign of business processes by making full use of voice input and AI.

[0879] (Application example 1)

[0880] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0881] In production sites, optimizing production processes is difficult, and there are many inefficient manual tasks and wasteful operations. To solve these problems, there is a need for a support system that allows on-site operators and engineers to intuitively optimize production processes.

[0882] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0883] In this invention, the server includes a speech recognition unit, a unit for converting speech data into text data, a unit for analyzing the text data and generating a workflow diagram, a unit for displaying the generated workflow diagram, a unit for receiving additional information from a user, a unit for modifying the workflow diagram using the additional information, a unit for extracting issues related to the workflow and presenting solutions, a unit for converting the generated workflow diagram, issues, and solutions into slides and videos, a unit for providing the slides and videos to the user, a unit for automatically generating a production process flow diagram, extracting issues, and proposing solutions based on a voice-input explanation for optimizing the production process flow, and a unit for visualizing the results including the issues and solutions as industrial slides. This enables operators and engineers to optimize the production process through intuitive voice operations.

[0884] A "voice recognition means" is a device or software system for converting voice data into text data.

[0885] "Means for converting voice data into text data" refers to the technology or process for converting recorded voice data into text data as character information.

[0886] The "means for analyzing text data and generating a workflow diagram" refers to an algorithm or system for analyzing the converted text data and generating a workflow diagram that illustrates the flow and procedures of a business process.

[0887] The "means for displaying the generated workflow diagram" refers to a device or software for visually displaying to the user the workflow diagram generated based on the analysis.

[0888] A "means for receiving additional information from a user" is an interface or system for receiving supplemental information or correction requests provided by a user.

[0889] The "means for modifying a workflow diagram using additional information" refers to a system or algorithm for modifying an existing workflow diagram based on additional information received from a user.

[0890] The "means for extracting issues related to business flow and proposing solutions" is an analysis system that identifies potential problems and areas for improvement from the generated business flow diagram and proposes solutions based on that.

[0891] The "means for converting the generated business flow diagram, issues, and solutions into slides and videos" refers to a tool or system for outputting the business flow diagram, issues, and solutions as presentation slides and videos.

[0892] The "means for providing slides and videos to users" refers to a device or platform for providing the generated slides and videos to users in a viewable form.

[0893] "Means for automatically generating a production process flow chart based on voice-input explanations in order to optimize the production process flow, extracting issues, and proposing solutions" is a system that automatically generates a production process flow chart based on voice input from the user, identifies points that need improvement from the chart, and proposes solutions.

[0894] "Means for visualizing results including problems and solutions as industrial slides" refers to tools and systems for visually displaying extracted problems and proposed solutions as industrial presentation slides.

[0895] To realize a system based on this invention, multiple steps must be performed: voice input, voice recognition, text analysis, business flow diagram generation, problem extraction, solution presentation, and final slide generation. The hardware and software configuration and processing required to realize this system are described in detail below.

[0896] Hardware Configuration

[0897] 1. Smartphone: This device is used by the user for voice input.

[0898] 2. Server: A central computer that processes and analyzes audio data.

[0899] Software Configuration

[0900] 1. Speech recognition software: For example, use the SpeechRecognition library or Google's speech recognition API.

[0901] 2. Text analysis and generative AI models: Use OpenAI's API.

[0902] 3. Presentation generation tool: Uses the Python-pptx library.

[0903] Processing Description

[0904] 1. Voice Input and Recognition

[0905] The user verbally explains the production process using the smartphone's microphone, for example, "First, we receive the raw materials, then we conduct a quality inspection, then we start assembly. Finally, we inspect the products and prepare them for shipping."

[0906] The terminal records this voice and sends it to the server.

[0907] 2. Voice Recognition

[0908] The server converts the voice data into text data using a speech recognition engine (for example, Google's speech recognition API). For example, the server obtains text data such as "First, we receive the raw materials, then we conduct a quality inspection, then we start the assembly process. Finally, we inspect the products and prepare them for shipping."

[0909] 3. Creating a production process flow

[0910] The server uses OpenAI's generative AI model to analyze the received text data and generate a production process flow diagram. The prompt is "Generate a production process flow diagram based on the following sentence: First, receive the raw materials, then perform quality inspection, then start assembly, and finally inspect and prepare for shipment."

[0911] The AI ​​model outputs a flow diagram of "Acceptance → Quality inspection → Assembly → Inspection → Preparation for shipment."

[0912] 4. Modifying the flow diagram

[0913] If the generated flow diagram contains any deficiencies or errors, the server notifies the user. The user can then provide additional explanations by voice and send them to the server via their terminal. For example, the user can provide additional information such as, "Some parts need to be re-inspected after assembly."

[0914] The server then performs speech recognition on the speech and converts it into text. The additional information is reflected in the flow diagram, generating a revised diagram.

[0915] 5. Identifying issues and presenting solutions

[0916] The server then uses the AI ​​model again to extract issues from the generated production process flow diagram, such as delays in assembly work.

[0917] Then, based on that, they present solutions, such as proposing "prevent delays by promoting automation of assembly work."

[0918] 6. Slide and video generation

[0919] The server uses Python-pptx to compile the generated flow diagrams, issues, and solutions into slides, which can then be output as videos.

[0920] Finally, the product is provided to the user through a terminal.

[0921] Specific examples

[0922] For example, if a user describes, "First, we receive the raw materials, then we perform quality inspection, then we perform assembly work, and finally we inspect the products and prepare them for shipping," the system will operate as follows:

[0923] 1. Speech recognition of user's explanation

[0924] 2. Convert to text data

[0925] 3. Generate a production process flow diagram: "Receipt → Quality inspection → Assembly → Inspection → Shipping preparation"

[0926] 4. Receive additional information from the user and modify the flow diagram as needed.

[0927] 5. Identify issues and present solutions

[0928] 6. Organize the final results in slide format and provide them to the user

[0929] This system allows for efficient and intuitive optimization of production processes simply by using voice input via a smartphone.

[0930] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0931] Step 1:

[0932] The user verbally describes the production process using the smartphone's microphone. For example, "First, we receive the raw materials, then we conduct quality inspections, then we perform assembly work, and finally we inspect the products and prepare them for shipment." The input is voice data.

[0933] Step 2:

[0934] The terminal records the voice input by the user and sends it to the server. The input is voice data, and the output is a data file sent to the server.

[0935] Step 3:

[0936] The server uses a speech recognition engine (for example, Google's speech recognition API) to convert the received voice data into text data. Specifically, the speech recognition model analyzes the voice signal and generates a corresponding text string. The input is the voice data, and the output is the text data: "First, receive the raw materials, then perform quality inspection, then perform assembly work, and finally inspect and prepare for shipment."

[0937] Step 4:

[0938] The server uses a generative AI model (for example, OpenAI's GPT-3) to analyze the obtained text data and generate a production process flow diagram. The prompt is input to the model in the form of "Generate a production process flow diagram based on the following text: First, receive the raw materials, then perform quality inspection, then perform assembly work. Finally, perform inspection and prepare for shipment." The generative AI model analyzes the text and generates a flow diagram of "Receipt → Quality inspection → Assembly → Inspection → Prepare for shipment." The input is text data, and the output is a production process flow diagram.

[0939] Step 5:

[0940] The server displays the generated flow diagram on the terminal, allowing the user to visually confirm it. The input is the production process flow diagram, and the output is the flow diagram displayed on the terminal.

[0941] Step 6:

[0942] If there are any omissions or errors in the flow diagram, the user can provide additional explanations by voice input, for example, "Some parts need to be re-inspected after assembly." The input is voice data.

[0943] Step 7:

[0944] The device records additional audio data and sends it back to the server. The input is the audio data and the output is the data file sent to the server.

[0945] Step 8:

[0946] The server again uses the speech recognition engine to convert the additional voice data into text data. Specifically, the speech recognition model analyzes the voice signal and generates text such as, "Some parts need to be re-inspected after assembly." The input is voice data, and the output is text data.

[0947] Step 9:

[0948] The server modifies the workflow diagram based on the additional text data. For example, it modifies it to the format "Acceptance → Quality Inspection → Assembly → Re-inspection → Inspection → Shipping Preparation." The input is text data, and the output is the modified production process flow diagram.

[0949] Step 10:

[0950] The server extracts issues from the revised flow diagram and proposes solutions using a generative AI model. For example, a delay in assembly work is extracted as an issue, and "automation of assembly work" is proposed as a solution. The input is the revised production process flow diagram, and the output is the issue and solution.

[0951] Step 11:

[0952] The server uses Python-pptx to compile the generated production process flow diagrams, problems, and solutions into slides, which can then be presented to users in a visually easy-to-understand format. The input is text data containing problems and solutions, and the output is presentation slides.

[0953] Step 12:

[0954] Finally, the terminal provides the generated slides to the user, who can visually review them and take action to improve the production process if necessary. The input is the presentation slides, and the output is the visual information provided to the user.

[0955] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0956] This invention is a system that supports business process redesign (BPR) in a conversational format using natural language, and by combining speech recognition technology, text analysis technology, a generative AI model, and an emotion engine, it generates and modifies business flow diagrams, extracts issues, presents solutions, and makes adjustments based on the user's emotions. This system operates mainly on a server, terminals, and users.

[0957] First, the user starts the voice input application and verbally explains the business flow using the device. For example, they might say, "In this department, we first receive the application form, check the contents, and then circulate it to our superiors." The device records this and sends it to the server as voice data.

[0958] Next, the server converts the received voice data into text data using a speech recognition device. Based on this converted text data, an initial workflow diagram is generated. The generated workflow diagram is then displayed on the user's device.

[0959] The server also uses an emotion engine to recognize emotions from the user's voice. For example, if the user is nervous or excited during the explanation, the server can detect the user's emotional state and adjust the generation or modification of the workflow diagram accordingly.

[0960] If the displayed workflow diagram contains any branching points or missing information, the server detects this and notifies the user. The user then provides additional verbal explanation, which is recorded on the device and sent to the server. The server then recognizes the speech again, converts it into text data, and modifies the workflow diagram.

[0961] The server can also use an emotion engine to continue the dialogue depending on the user's emotional state: for example, if the user is feeling stressed, it can provide feedback to promote relaxation.

[0962] The server also uses the problem extraction engine to extract related issues from the generated business flow diagram. It then generates solutions based on these and presents them to the user. For example, specific proposals such as "We propose the introduction of an electronic approval system to speed up approvals."

[0963] Finally, the server creates slides containing the business flow diagram, issues, and solutions, and converts them into video format. The generated slides and videos are then provided to the user via their device. The user can visually check this information and identify areas for improvement and countermeasures for the business process.

[0964] Specific examples

[0965] 1. Voice Input and Recognition

[0966] The user verbally explains the business flow to the terminal, saying, "First, receive the application form, check the contents, and then forward it to your supervisor." The terminal records the voice and sends it to the server. The server uses a voice recognition engine to convert the voice data into text data that says, "First, receive the application form, check the contents, and then forward it to your supervisor."

[0967] 2. Generate a workflow diagram

[0968] The server analyzes the text data and generates a workflow diagram of "Accept → Confirm → Circulate." It then uses an emotion engine to analyze the user's emotional state and makes adjustments according to their emotions. The server then presents the generated workflow diagram to the user.

[0969] 3. Modifying the Workflow Diagram

[0970] The user verbally explains the additional information, saying, "After circulation, once my superior approves, I will pass it on to the next department." The device records the additional voice and sends it to the server. The server recognizes this and modifies the workflow diagram to "Accept → Confirm → Circulate → Approve → Pass on to the next department." The emotion engine also analyzes the user's emotional state and adjusts the feedback when providing additional explanations.

[0971] 4. Presenting the problem and solution

[0972] The server extracts the issue of "approval delays" and generates a solution: "Propose the introduction of an electronic approval system to speed up approvals." The server converts the final business flow diagram, issue, and solution into slides and videos and provides them to the user. The user reviews the provided slides and videos and identifies areas for improvement in the business process and measures to be taken.

[0973] In this way, the present invention provides a system that efficiently and intuitively supports a series of processes for implementing BPR in a conversational manner using natural language, including adjustments based on emotions.

[0974] The processing flow will be explained below.

[0975] The program processing will be specifically explained below by dividing it into steps.

[0976] ---

[0977] Step 1:

[0978] The user starts the voice input application on the device and verbally explains the business flow, for example, "In this department, we first receive the application form, check its contents, and then forward it to the superior."

[0979] Step 2:

[0980] The device records the user's voice and, when the recording is finished, sends the voice data to the server.

[0981] Step 3:

[0982] The server sends the received voice data to a voice recognition engine, which converts the voice data into text data, which is then stored in a database.

[0983] Step 4:

[0984] The server analyzes the stored text data and uses a contextual analysis engine to extract each step of the process.

[0985] Step 5:

[0986] The server automatically generates an initial workflow diagram based on the extracted business steps, and the generated workflow diagram is displayed on the user's device.

[0987] Step 6:

[0988] The server analyzes the business flow diagram to determine whether there are any branching points or missing information. If any branching points or missing information are detected, the server notifies the user.

[0989] Step 7:

[0990] The user verbally provides additional explanations about branching points and missing information through a voice input application, for example, "After circulation, once the supervisor approves, we will pass it on to the next department."

[0991] Step 8:

[0992] The device records additional audio and sends it back to the server.

[0993] Step 9:

[0994] The server recognizes the received additional voice data and converts it into text data.Then, it modifies the workflow diagram based on the received additional information.

[0995] Step 10:

[0996] The server uses an emotion engine to recognize emotions from the user's voice. For example, if the user is nervous or excited, the server detects that emotional state.

[0997] Step 11:

[0998] The server adjusts the generation and modification of the workflow diagram based on the extracted emotional state. For example, if the user is nervous, the server will adjust the feedback to be gentler.

[0999] Step 12:

[1000] The server uses an issue extraction engine to identify potential problems and bottlenecks based on the business flow diagram.

[1001] Step 13:

[1002] The server automatically generates solutions based on the extracted issues, making specific proposals such as "Propose the introduction of an electronic approval system to speed up approvals."

[1003] Step 14:

[1004] The server compiles the final business flow diagram, issues, and solutions into slides, which are then converted into video format.

[1005] Step 15:

[1006] The server transmits the generated slides and videos to the user's terminal.

[1007] Step 16:

[1008] The user reviews the provided slides and videos to identify areas for improvement and measures for business processes.

[1009] ---

[1010] These are the specific processing steps of the program. This process allows users to review and improve their business processes efficiently simply by explaining the workflow in natural language. The emotion engine provides responses and feedback based on the user's emotions, supporting more effective BPR.

[1011] Example 2

[1012] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1013] In conventional business process redesign (BPR) systems, when generating a workflow diagram from voice input, the process proceeds without taking into account the user's emotional state, which can lead to stress and misunderstandings. Another issue is the inefficiency of the problem extraction and solution presentation processes. Furthermore, there is a lack of a way to present the generated information visually in an easy-to-understand manner.

[1014] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a voice recognition means, a means for converting voice data into text data, and a means for recognizing the user's emotions and adjusting the generation and modification of a workflow diagram based on the emotions. This enables flexible responses according to the user's emotional state, and enables efficient generation and modification of a workflow diagram while reducing stress and misunderstandings.

[1015] "Speech recognition means" refers to a device or program for converting voice data into text data.

[1016] "Means for converting voice data into text data" refers to technology that analyzes recorded voice data and converts its contents into text information.

[1017] "Means for analyzing text data and generating business flow diagrams" refers to technology for extracting business procedures and flows from text data and generating a flowchart that visually represents them.

[1018] The "means for displaying the generated workflow diagram" refers to a device or program for displaying the generated workflow diagram on a terminal display or monitor.

[1019] The "means for receiving additional information from the user" is an interface that allows the user to provide further supplementary explanations or corrections to the workflow diagram verbally or by other means.

[1020] The "means for modifying a workflow diagram using additional information" is a technique for updating or modifying an existing workflow diagram based on additional information received from a user.

[1021] "Means for extracting issues related to business flow and proposing solutions" is a technology for finding problems in the generated business flow diagram and proposing specific solutions to them.

[1022] "Means for converting the generated business flow diagram, issues, and solutions into slides and videos" refers to technology for converting the generated business flow diagram, issues, and solutions into presentation slides or video format in order to present them in a visually easy-to-understand manner.

[1023] The "means for providing slides and videos to the user" refers to a technique for visually presenting the generated slides and videos to the user.

[1024] "Means for recognizing a user's emotions and adjusting the generation and modification of a workflow diagram based on those emotions" is a technology that analyzes the user's emotional state from their voice and actions, and adjusts the progress of the generation and modification of a workflow diagram based on those emotions.

[1025] This invention is a system that supports business process redesign (BPR) in a conversational format using natural language. This system combines speech recognition technology, text analysis technology, a generative AI model, and an emotion engine to generate and modify business flow diagrams, identify issues, present solutions, and make adjustments based on the user's emotions. This system operates primarily on a server, terminals, and users.

[1026] First, the user starts the voice input application and verbally explains the business flow using the device. For example, they might say, "In this department, we first receive the application form, check the contents, and then circulate it to our superiors." The device records this and sends it to the server as voice data.

[1027] Next, the server uses a speech recognition tool to convert the received voice data into text data. Based on this converted text data, an initial workflow diagram is generated. Using a generative AI model, the steps and flow of the business are extracted from the text data, and a flowchart that visually represents them is generated. The generated workflow diagram is then displayed on the user's device.

[1028] The server also uses an emotion engine to recognize emotions from the user's voice. For example, if the user is nervous or excited during the explanation, the server can detect that emotional state and adjust the generation or modification of the workflow diagram accordingly.

[1029] If the displayed workflow diagram contains any branching points or missing information, the server detects this and notifies the user. The user then provides additional verbal explanation, which is recorded on the device and sent to the server. The server then recognizes the speech again, converts it into text data, and modifies the workflow diagram.

[1030] The server also uses a problem extraction engine to extract related issues from the generated business flow diagram. Based on this, it generates solutions and presents them to the user. For example, specific proposals such as "We propose the introduction of an electronic approval system to speed up approvals" are included.

[1031] Finally, the server creates slides containing the business flow diagram, issues, and solutions, and converts them into video format. The generated slides and videos are then provided to the user via their device. The user can visually check this information and identify areas for improvement and countermeasures for the business process.

[1032] As a specific example, the following steps can be mentioned.

[1033] 1. Voice Input and Recognition

[1034] The user verbally explains the business flow to the terminal, saying, "First, receive the application form, check the contents, and then forward it to your supervisor." The terminal records the voice and sends it to the server. The server uses a voice recognition engine to convert the voice data into text data that says, "First, receive the application form, check the contents, and then forward it to your supervisor."

[1035] 2. Generate a workflow diagram

[1036] The server analyzes the text data and generates a workflow diagram of "Accept → Confirm → Circulate." It then uses an emotion engine to analyze the user's emotional state and makes adjustments according to their emotions. The server then presents the generated workflow diagram to the user.

[1037] 3. Modifying the Workflow Diagram

[1038] The user verbally explains the additional information, saying, "After circulation, once my superior approves, I will pass it on to the next department." The device records the additional voice and sends it to the server. The server recognizes this and modifies the workflow diagram to "Accept → Confirm → Circulate → Approve → Pass on to the next department." The emotion engine also analyzes the user's emotional state and adjusts the feedback when providing additional explanations.

[1039] 4. Presenting the problem and solution

[1040] The server extracts the issue of "approval delays" and generates a solution: "Propose the introduction of an electronic approval system to speed up approvals." The server converts the final business flow diagram, issue, and solution into slides and videos and provides them to the user. The user reviews the provided slides and videos and identifies areas for improvement in the business process and measures to be taken.

[1041] As described above, the present invention is a system that implements BPR in a conversational format using natural language, and provides efficient and intuitive support, including adjustments based on emotions.

[1042] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1043] Step 1:

[1044] The user starts a voice input application and verbally explains the business flow. For example, "In this department, we first receive the application form, check the contents, and then circulate it to our superiors." The input is voice data, and the output is a recorded voice file. The device records the user's voice and saves the voice data.

[1045] Step 2:

[1046] The device sends the recorded audio file to the server. The input is the recorded audio file, and the output is the audio data transferred to the server. The device converts the audio file to an appropriate format (e.g., WAV format) and sends it to the server via an HTTP request.

[1047] Step 3:

[1048] The server converts the received voice data into text data using a voice recognition engine (e.g., Google Speech-to-Text API). The input is voice data, and the output is text data. The server passes the voice file to the voice recognition engine and obtains the converted text data.

[1049] Step 4:

[1050] The server uses a generative AI model to extract the workflow from the converted text data and generate a workflow diagram. The input is text data, and the output is workflow diagram data. The server analyzes the text and extracts workflow steps such as "accept → confirm → circulate" based on the content.

[1051] Step 5:

[1052] The server uses an emotion engine to analyze the user's emotions. The input is voice data and text data, and the output is emotional state data. The server passes the voice data to the emotion engine for analysis and obtains results such as whether the user is nervous or excited.

[1053] Step 6:

[1054] The server displays the generated workflow diagram on the user's terminal. The input is the workflow diagram data, and the output is a visual display on the terminal. The server sends the workflow diagram to the user's terminal, and the terminal displays it on the screen.

[1055] Step 7:

[1056] The user verbally provides supplementary information to add explanations or make corrections to the business flow diagram. For example, the user might say, "After circulation, once my superior approves, I will pass it on to the next department." The input is additional voice data, and the output is the recorded supplementary voice data.

[1057] Step 8:

[1058] The device re-records the additional audio data and sends it to the server. The input is the additional audio data and the output is the supplemental audio data transferred to the server. The device again converts the audio file into an appropriate format and sends it to the server.

[1059] Step 9:

[1060] The server recognizes the additional voice data and converts it into text data. The input is the additional voice data and the output is the additional text data. The server uses a voice recognition engine to convert the additional explanation into text data.

[1061] Step 10:

[1062] The server modifies the workflow diagram based on the added text data. The input is the additional text data and the existing workflow diagram, and the output is the modified workflow diagram. Based on the additional explanation, the server updates the workflow diagram to "Accept → Confirm → Circulate → Approve → Send to next department."

[1063] Step 11:

[1064] The server uses the problem extraction engine to extract issues from the generated business flow diagram and presents solutions. The input is the modified business flow diagram, and the output is data on the issues and solutions. For example, it extracts the issue of "approval delays" and generates a solution such as "proposing the introduction of an electronic approval system."

[1065] Step 12:

[1066] The server creates slides based on the business flow diagram, issues, and solutions, and converts them into presentation slides and video formats. The input is the business flow diagram, issues, and solution data, and the output is the generated slides and videos. The server uses a slide creation tool to generate visually easy-to-understand slides and videos.

[1067] Step 13:

[1068] The server provides the generated slides and videos to the user's terminal. The input is slide and video data, and the output is a visual display on the user's terminal. The server sends the slides and videos to the user's terminal, and the user visually checks them.

[1069] The above are the processing steps of the program of this system, and summarize the specific operations, inputs, and outputs performed at each step.

[1070] (Application example 2)

[1071] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1072] Traditional business process redesign (BPR) is often highly manual, inefficient, and places a heavy burden on employees, especially those working in the field. It is also difficult to consider employees' emotional states, which can negatively impact stress and work efficiency. This invention aims to solve these problems and provide a system for redesigning business processes more efficiently and intuitively.

[1073] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1074] In this invention, the server includes a speech recognition unit, a means for converting speech data into text data, a means for analyzing the text data and generating a workflow diagram, a means for displaying the generated workflow diagram, a means for receiving additional information from a user, a means for modifying the workflow diagram using the additional information, a means for extracting issues related to the workflow and presenting solutions, a means for converting the generated workflow diagram, issues, and solutions into slides and videos, a means for providing the slides and videos to the user, an emotion recognition unit for analyzing the user's emotions from speech and providing feedback based on the emotional state, and a means for recording the user's voice when the user gives instructions on a business process and generating and modifying a workflow diagram. This allows on-site employees to intuitively generate and modify a workflow diagram simply by verbally explaining the business process, and provides optimal feedback according to the employee's emotional state, thereby improving work efficiency and reducing stress.

[1075] "Speech recognition means" is a technology that acquires and analyzes what a user says orally as digital voice data.

[1076] The "means for converting voice data into text data" refers to a technology for analyzing acquired digital voice data and converting it into corresponding text data.

[1077] The "means for analyzing text data and generating a business flow diagram" is a technology that automatically generates a flow diagram that visualizes the flow of business operations based on converted text data.

[1078] The "means for displaying the generated workflow diagram" is a technique for visually presenting the generated workflow diagram on the user's terminal.

[1079] The "means for receiving additional information from the user" refers to a technique for obtaining supplementary or corrective information provided by the user.

[1080] The "means for modifying a workflow diagram using additional information" is a technique for updating an existing workflow diagram based on additional information received from a user.

[1081] "Means for extracting issues related to business flow and proposing solutions" is a technology that finds problems in the generated business flow diagram and proposes solutions to those problems.

[1082] "Means for converting the generated business flow diagram, issues, and solutions into slides and videos" refers to a technology for converting business flow diagrams, extracted issues, and their solutions into visual slide and video formats.

[1083] The "means for providing slides and videos to users" refers to a technique for displaying or distributing the generated slides and videos to users.

[1084] "Emotion recognition means" is a technology that analyzes emotions from the user's voice and identifies their emotional state.

[1085] "Means for recording the voice of a user giving instructions on a business process and generating or modifying a business flow diagram" refers to a technology that records the user's verbal explanation of a business process and creates or modifies a business flow diagram based on that data.

[1086] This invention is a system that supports business process redesign (BPR) in a conversational format using natural language. By combining speech recognition technology, text analysis technology, a generative AI model, and an emotion engine, the system generates and modifies business flow diagrams, extracts issues, presents solutions, and makes adjustments based on the user's emotions. This system is mainly operated by a server, terminals, and users.

[1087] System configuration

[1088] 1. Voice recognition: The user verbally explains the business flow and the voice is captured as digital data. This is done using a smartphone with a built-in microphone.

[1089] 2. A method for converting voice data to text data: Analyze the voice data and convert it to the corresponding text data. This conversion is performed using the Python speech_recognition library.

[1090] 3. A method for analyzing text data and generating workflow diagrams: A workflow diagram is automatically generated based on the converted text data. A text generation model in Hugging Face's Transformers library is used.

[1091] 4. A means for displaying the generated workflow diagram: The generated workflow diagram is visually displayed on the user's device, using the smartphone screen.

[1092] 5. Means of receiving additional information from the user: A function for obtaining supplementary or corrective information provided by the user.

[1093] 6. A method for modifying a workflow diagram using additional information: Update an existing workflow diagram based on additional information from the user. This also uses speech recognition and text analysis technologies.

[1094] 7. A means to extract issues related to business processes and propose solutions: Identify problems in the generated business process diagram and propose solutions to those problems. This function uses a generative AI model.

[1095] 8. A means of converting generated workflow diagrams, issues, and solutions into slides and videos: A function to convert workflow diagrams and extracted issues and solutions into visual slide and video formats.

[1096] 9. Means of providing slides and videos to users: The ability to display or distribute the generated slides and videos to users.

[1097] 10. Emotion Recognition: Analyzes emotions from the user's voice and identifies their emotional state. This analysis is done using TextBlob and a customized emotion recognition engine, EmotionRecognizer.

[1098] 11. When giving instructions on a business process, a means for recording the voice and generating and modifying a business flow diagram: A user's verbal explanation of a business process is recorded, and a business flow diagram is created and modified based on that data.

[1099] Specific examples

[1100] 1. Voice Input and Recognition

[1101] The user verbally explains the workflow to the terminal, saying, "In this store, first we place products on the shelves, then we check the inventory and take our turn at the cash register." The terminal records the voice and sends it to the server, which uses a speech recognition engine to convert the voice data into text data.

[1102] 2. Generate a workflow diagram

[1103] The server analyzes the text data and generates a workflow diagram of "Shelf placement → Inventory check → Register shift." It then uses an emotion engine to analyze the user's emotional state and makes adjustments according to their emotions. The server then presents the generated workflow diagram to the user.

[1104] 3. Modifying the Workflow Diagram

[1105] The user verbally explains additional information, saying, "After the cashier changes, we will close the register." The device records the additional voice and sends it to the server. The server recognizes this and modifies the workflow diagram to "Shelf arrangement → Inventory check → Cashier change → Close the register." The emotion engine also analyzes the user's emotional state and adjusts the feedback when providing additional explanations.

[1106] 4. Presenting the problem and solution

[1107] The server extracts the issue that "cash register closing work is often delayed" and generates a solution that "proposes the introduction of an automatic cash register closing system." The server converts the final business flow diagram, issue, and solution into slides and videos and provides them to the user. The user reviews the provided slides and videos and understands the areas for improvement and countermeasures for the business process.

[1108] Prompt Sentence Examples

[1109] Current work flow: Place products on shelves, check inventory, and shift to the cash register.

[1110] Please give me some suggestions for improving this workflow. Please tell me specifically what to do, including the solution.

[1111] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1112] Step 1:

[1113] The user verbally explains the workflow, saying, "In this store, first we place products on the shelves, then we check the inventory and take turns at the cash register." The terminal records the voice and saves it as voice data. The input is the user's voice, and the output is voice data.

[1114] Step 2:

[1115] The server converts the voice data received from the terminal into text data using a voice recognition method. Specifically, it uses the Python speech_recognition library to analyze the voice data and generate Japanese text data. The input is voice data and the output is text data.

[1116] Step 3:

[1117] The server analyzes the text data and generates a workflow diagram. Using the text generation model in Hugging Face's Transformers library, a workflow diagram of "Shelf placement → Stock check → Register shift" is automatically generated from the text data. The input is text data, and the output is a workflow diagram.

[1118] Step 4:

[1119] The generated workflow diagram is sent to the user's terminal and displayed. The terminal visually presents the workflow diagram to the user. The input is the workflow diagram, and the output is the workflow diagram displayed on the terminal screen.

[1120] Step 5:

[1121] The user verbally explains changes to the workflow and additional information, such as "After the cashier change, we will perform the cashier closing work." The terminal records the additional voice and sends it to the server. The input is the user's voice, and the output is voice data.

[1122] Step 6:

[1123] The server converts the additional voice data into text data again using a voice recognition means, with the input being voice data and the output being text data.

[1124] Step 7:

[1125] The server reflects the additional text data in the original workflow diagram and modifies the workflow diagram to "Shelf placement → Stock check → Register shift → Cashier closing work." The input is the new text data and the existing workflow diagram, and the output is the modified workflow diagram.

[1126] Step 8:

[1127] The server extracts issues from the workflow diagram and proposes solutions using a generative AI model. For example, it extracts the issue "cash register closing work is often delayed" and proposes "the introduction of an automatic cash register closing system." The input is the workflow diagram, and the output is the issue and solution.

[1128] Step 9:

[1129] The generated workflow diagram, issues, and solutions are converted into slides and videos and sent to the user's device. The device visually presents the slides and videos to the user. The input is the workflow diagram, issues, and solutions, and the output is the slides and videos.

[1130] Step 10:

[1131] The server uses emotion recognition to analyze the emotions in the user's voice and provides feedback based on the user's emotional state. For example, if the user is feeling stressed, it generates feedback to encourage relaxation. The input is the user's voice and the output is feedback.

[1132] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1133] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1134] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1135] [Fourth embodiment]

[1136] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1137] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1138] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1139] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1140] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1141] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1142] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1143] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1144] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1145] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1146] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1147] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1148] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1149] This invention is a system that supports business process redesign (BPR) in a conversational format using natural language, and mainly uses speech recognition technology, text analysis technology, and generative AI models to automatically generate and modify business flow diagrams, identify issues, and present solutions. This system operates mainly on a server, terminals, and users.

[1150] First, the user starts the voice input application and verbally explains the business flow using the device. For example, the user might explain a specific business process, such as, "In this department, we first receive the application form, check the contents, and then circulate it to our superiors." The device then records this and sends it to the server as voice data.

[1151] Next, the server uses a speech recognition means to convert the received voice data into text data. Based on this text data, an initial workflow diagram is generated. The generated workflow diagram is then displayed to the user via their terminal.

[1152] If the displayed workflow diagram has any branching points or missing information, the server detects this and notifies the user. The user then provides additional verbal explanation, which is then re-recorded on the device and sent to the server. The server then recognizes this speech again, converts it into text data, and corrects the workflow diagram.

[1153] Furthermore, the server uses a problem extraction method to extract related issues from the generated business flow diagram, identifying potential problems and areas for improvement. Based on this, it generates solutions and presents them to the user. These solutions include specific proposals, such as "We propose the introduction of an electronic approval system to speed up approvals."

[1154] Finally, the server compiles the final workflow diagram, problems, and solutions into slides, and then converts them into video format. These outputs are provided to the user via their device, allowing them to visually confirm the information.

[1155] Specific examples

[1156] 1. Voice Input and Recognition

[1157] The user verbally explains the business flow to the terminal, saying, "First, I will receive the application form, check the contents, and then pass it on to my supervisor."

[1158] The device records the audio and sends it to the server.

[1159] The server uses a voice recognition engine to convert the voice data into text data such as "First, I will receive the application form, check the contents, and then forward it to my supervisor."

[1160] 2. Generate a workflow diagram

[1161] The server analyzes the text data and generates a business flow diagram of "Accept → Confirm → Circulate."

[1162] The server presents the generated business flow diagram to the user.

[1163] 3. Modifying the Workflow Diagram

[1164] The user verbally explains the additional information, saying, "After circulation, once my superior approves, I will pass it on to the next department."

[1165] The device records additional audio and sends it to the server.

[1166] The server recognizes this voice and modifies the workflow diagram to "Accept → Confirm → Circulate → Approve → Send to next department."

[1167] 4. Presenting the problem and solution

[1168] The server extracts the problem that "approvals are prone to delays" and generates a solution that "proposes the introduction of an electronic approval system to speed up approvals."

[1169] The server converts the final business flow diagram, problems, and solutions into slides and videos and provides them to the user.

[1170] In this way, the present invention is a system that efficiently and intuitively supports a series of processes for implementing BPR in a conversational format using natural language.

[1171] The processing flow will be explained below.

[1172] The program processing will be specifically explained below by dividing it into steps.

[1173] ---

[1174] Step 1:

[1175] The user starts the voice input application on the device and verbally explains the business flow, for example, "In this department, we first receive the application form, check its contents, and then forward it to the superior."

[1176] Step 2:

[1177] The device records the user's voice and, when the recording is finished, sends the voice data to the server.

[1178] Step 3:

[1179] The server sends the received voice data to a voice recognition engine, which converts the voice data into text data, which is then stored in a database.

[1180] Step 4:

[1181] The server analyzes the stored text data and uses a contextual analysis engine to extract each step of the process.

[1182] Step 5:

[1183] The server automatically generates an initial workflow diagram based on the extracted business steps, and presents the generated workflow diagram to the user.

[1184] Step 6:

[1185] The server analyzes the business flow diagram to determine whether there are any branching points or missing information. If any branching points or missing information are detected, the server notifies the user.

[1186] Step 7:

[1187] The user verbally provides additional explanations about branching points and missing information through a voice input application, for example, "After circulation, once the supervisor approves, we will pass it on to the next department."

[1188] Step 8:

[1189] The device records additional audio and sends it back to the server.

[1190] Step 9:

[1191] The server recognizes the received additional voice data and converts it into text data.Then, it modifies the workflow diagram based on the received additional information.

[1192] Step 10:

[1193] The server uses an issue extraction engine to identify potential problems and bottlenecks based on the business flow diagram.

[1194] Step 11:

[1195] The server automatically generates solutions based on the extracted issues, making specific proposals such as "Propose the introduction of an electronic approval system to speed up approvals."

[1196] Step 12:

[1197] The server compiles the final business flow diagram, issues, and solutions into slides, which are then converted into video format.

[1198] Step 13:

[1199] The server transmits the generated slides and videos to the user's terminal.

[1200] Step 14:

[1201] The user reviews the provided slides and videos to identify areas for improvement and measures for business processes.

[1202] ---

[1203] These are the specific processing steps of the program. This process allows users to review and improve their business processes efficiently simply by explaining the workflow in natural language.

[1204] Example 1

[1205] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1206] In conventional business process redesign (BPR), understanding the current state of business processes, identifying issues, and proposing improvement measures are often done manually or on paper, requiring time and effort. Furthermore, when using only voice input, the voice data is not converted into text, and the creation and revision of business flow diagrams are not automated, making it difficult to efficiently redesign business processes. Furthermore, because the extraction of issues and the generation of solutions are also manual, they can be subjective and lack comprehensiveness.

[1207] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1208] In this invention, the server includes a speech recognition unit, a unit for converting speech data into text data, and a unit for analyzing the text data and generating a workflow diagram, which makes it possible to generate and modify workflows, extract problems, and present solutions simply by voice input.

[1209] "Speech recognition means" refers to technical devices or software for analyzing input voice data and converting it into text data.

[1210] "Means for converting voice data into text data" refers to technical devices or software that use speech recognition means to analyze voice data and convert it into human-readable text data.

[1211] "Means for analyzing text data and generating a workflow diagram" refers to technical devices or software that analyzes the content of text data and automatically generates a workflow diagram that visually shows business procedures.

[1212] "Means for presenting the generated workflow diagram" refers to technical devices or software that allow a user to visually confirm the generated workflow diagram.

[1213] "Means for receiving additional information from the user" means any technical device or software that accepts additional audio or text information provided by the user.

[1214] "Means for modifying a workflow diagram with additional information" refers to technical devices and software for updating and modifying an existing workflow diagram based on additional information provided by a user.

[1215] "Means for extracting issues related to business flow and proposing solutions" refers to technical devices and software for analyzing the generated business flow diagram, identifying potential problems, and generating and proposing solutions to those problems.

[1216] "Means for converting the generated workflow diagrams, problems, and solutions into a media format" refers to technical devices and software for converting the generated workflow diagrams, problems, and solutions into a visual format such as slides or videos.

[1217] "Means for providing users with data in media format" refers to technical devices and software that allow data converted into a media format to be displayed on a user's terminal or made available for download.

[1218] "Means for extracting issues and generating solutions using a technical model" refers to means for analyzing a business flow diagram using a specific technical model, extracting issues, and generating solutions.

[1219] "Means for proposing solutions to problems using generative AI models" refers to technical devices and software that use generative AI models to automatically generate specific solutions to problems and propose them to users.

[1220] This invention is a system that supports business process redesign (BPR) in a conversational format using natural language, and mainly uses speech recognition technology, text analysis technology, and generative AI models to automatically generate and modify business flow diagrams, identify issues, and present solutions. This system operates mainly on a server, terminals, and users.

[1221] First, the user launches a voice input application on their device. For example, a laptop or smartphone can be used. The user verbally explains the business flow into the device, for example, "First, receive the application form, check the contents, and then circulate it to your superior." The device then records this and sends it to the server as voice data. This process uses a high-performance microphone and a voice input application.

[1222] Next, the server uses a speech recognition engine (for example, Google Cloud Speech-to-Text) to convert the received voice data into text data. The speech recognition engine analyzes the voice data and converts it into an appropriate text format using a language model. An initial workflow diagram is generated based on this text data. The server uses a text analysis method (for example, a Python natural language processing library) to analyze the text data and generate an initial workflow diagram of "Accept → Confirm → Circulate."

[1223] The generated workflow diagram is displayed to the user via the terminal. For example, the workflow diagram is displayed graphically using a web browser or a dedicated viewer application. If the user determines that the displayed workflow diagram needs to be corrected, additional voice explanations are provided. For example, additional information is provided, such as "After circulation, once the supervisor approves, we will pass it on to the next department." The terminal records the additional voice and sends it back to the server.

[1224] The server performs voice recognition again and converts it into text data. Based on this, the workflow diagram is revised and updated to the following order: "Accept → Confirm → Circulate → Approve → Send to next department." Through this series of processes, an accurate workflow diagram is created that matches the user's intentions.

[1225] Furthermore, the server uses an issue extraction method to extract relevant issues from the generated business flow diagram and identifies potential problems and areas for improvement. For example, it automatically detects the issue of "approval delays." The server then uses a generative AI model (e.g., OpenAI GPT-4) to generate solutions such as "We propose the introduction of an electronic approval system to speed up approvals." An example of a prompt to be input to the generative AI model is "Please identify the problems with this business flow diagram and propose improvements."

[1226] Finally, the server converts the final business flow diagram, detected issues, and solutions into slides (e.g., PowerPoint) and video formats (e.g., MP4). These outputs are provided to users via their devices, allowing them to visually view the information. For example, they can view it on a laptop or smartphone screen. They can also be shared in a conference room using a projector.

[1227] This system allows users to redesign business processes efficiently and intuitively. By using natural language voice input, users can effortlessly generate and modify business flow diagrams, identify issues, and present solutions.

[1228] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1229] Step 1:

[1230] The user launches a voice input application on their device. At the user's workplace, they launch the application using a device such as a laptop or smartphone. The user then verbally explains the workflow. For example, they might say, "First, I will receive the application form, check its contents, and then circulate it to my superior." The device records this voice input through the microphone and saves it as voice data. This recorded voice data becomes the input.

[1231] Step 2:

[1232] The device sends the recorded voice data to the server over an internet connection. Specifically, the voice data is uploaded to the server using an HTTP POST request. The server receives the voice data and uses it as input for its speech recognition engine.

[1233] Step 3:

[1234] The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert the speech data into text data. Specifically, the server analyzes the speech data and applies a language model to generate text data. This converted text data is the output. The conversion from speech data to text data includes phonemic recognition and semantic analysis processes.

[1235] Step 4:

[1236] The server analyzes the text data and performs processing to generate an initial workflow diagram. Specifically, it uses natural language processing (NLP) technology to analyze the grammatical structure and meaning of the text. For example, from the expression "Receive application, confirm the contents, and circulate to superior," it generates a flow of "Accept → Confirm → Circulate." This workflow diagram is the output.

[1237] Step 5:

[1238] The server generates data to visualize the generated workflow diagram. For example, it generates image data in SVG or PNG format and sends it to the terminal. The terminal presents this workflow diagram to the user. The user visually checks the workflow diagram on the terminal screen.

[1239] Step 6:

[1240] If the user needs to provide additional explanations for the workflow diagram, they can use the voice input application again to provide the additional information verbally. For example, they can say, "After circulation, once the supervisor approves, we will pass it on to the next department." The device records this additional voice explanation and sends it back to the server. This additional voice data becomes the input.

[1241] Step 7:

[1242] The server converts the additional voice data into text data again using a speech recognition engine. The converted text data is analyzed and modifications are made to the existing workflow diagram. For example, the workflow diagram is updated to "Accept → Confirm → Circulate → Approve → Send to next department." This modified workflow diagram is output.

[1243] Step 8:

[1244] The server analyzes the revised workflow diagram and identifies potential problems using problem extraction methods. For example, it detects the problem of "approval is often delayed." This problem data is output.

[1245] Step 9:

[1246] The server uses a generative AI model (e.g., OpenAI GPT-4) to generate a solution to the problem. The prompt is "Please identify the problems in this workflow diagram and propose improvements," and the generated solution is output. For example, a proposed solution might be "We propose the introduction of an electronic approval system to speed up approvals."

[1247] Step 10:

[1248] The server converts the final business flow diagram, problems, and solutions into slide format (e.g., PowerPoint) and video format (e.g., MP4). These media format products are sent to the terminal. The terminal receives the sent data and provides it to the user. The user can visually check this information on the terminal screen.

[1249] In this way, the present invention is implemented as a system that efficiently supports the redesign of business processes by making full use of voice input and AI.

[1250] (Application example 1)

[1251] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1252] In production sites, optimizing production processes is difficult, and there are many inefficient manual tasks and wasteful operations. To solve these problems, there is a need for a support system that allows on-site operators and engineers to intuitively optimize production processes.

[1253] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1254] In this invention, the server includes a speech recognition unit, a unit for converting speech data into text data, a unit for analyzing the text data and generating a workflow diagram, a unit for displaying the generated workflow diagram, a unit for receiving additional information from a user, a unit for modifying the workflow diagram using the additional information, a unit for extracting issues related to the workflow and presenting solutions, a unit for converting the generated workflow diagram, issues, and solutions into slides and videos, a unit for providing the slides and videos to the user, a unit for automatically generating a production process flow diagram, extracting issues, and proposing solutions based on a voice-input explanation for optimizing the production process flow, and a unit for visualizing the results including the issues and solutions as industrial slides. This enables operators and engineers to optimize the production process through intuitive voice operations.

[1255] A "voice recognition means" is a device or software system for converting voice data into text data.

[1256] "Means for converting voice data into text data" refers to the technology or process for converting recorded voice data into text data as character information.

[1257] The "means for analyzing text data and generating a workflow diagram" refers to an algorithm or system for analyzing the converted text data and generating a workflow diagram that illustrates the flow and procedures of a business process.

[1258] The "means for displaying the generated workflow diagram" refers to a device or software for visually displaying to the user the workflow diagram generated based on the analysis.

[1259] A "means for receiving additional information from a user" is an interface or system for receiving supplemental information or correction requests provided by a user.

[1260] The "means for modifying a workflow diagram using additional information" refers to a system or algorithm for modifying an existing workflow diagram based on additional information received from a user.

[1261] The "means for extracting issues related to business flow and proposing solutions" is an analysis system that identifies potential problems and areas for improvement from the generated business flow diagram and proposes solutions based on that.

[1262] The "means for converting the generated business flow diagram, issues, and solutions into slides and videos" refers to a tool or system for outputting the business flow diagram, issues, and solutions as presentation slides and videos.

[1263] The "means for providing slides and videos to users" refers to a device or platform for providing the generated slides and videos to users in a viewable form.

[1264] "Means for automatically generating a production process flow chart based on voice-input explanations in order to optimize the production process flow, extracting issues, and proposing solutions" is a system that automatically generates a production process flow chart based on voice input from the user, identifies points that need improvement from the chart, and proposes solutions.

[1265] "Means for visualizing results including problems and solutions as industrial slides" refers to tools and systems for visually displaying extracted problems and proposed solutions as industrial presentation slides.

[1266] To realize a system based on this invention, multiple steps must be performed: voice input, voice recognition, text analysis, business flow diagram generation, problem extraction, solution presentation, and final slide generation. The hardware and software configuration and processing required to realize this system are described in detail below.

[1267] Hardware Configuration

[1268] 1. Smartphone: This device is used by the user for voice input.

[1269] 2. Server: A central computer that processes and analyzes audio data.

[1270] Software Configuration

[1271] 1. Speech recognition software: For example, use the SpeechRecognition library or Google's speech recognition API.

[1272] 2. Text analysis and generative AI models: Use OpenAI's API.

[1273] 3. Presentation generation tool: Uses the Python-pptx library.

[1274] Processing Description

[1275] 1. Voice Input and Recognition

[1276] The user verbally explains the production process using the smartphone's microphone, for example, "First, we receive the raw materials, then we conduct a quality inspection, then we start assembly. Finally, we inspect the products and prepare them for shipping."

[1277] The terminal records this voice and sends it to the server.

[1278] 2. Voice Recognition

[1279] The server converts the voice data into text data using a speech recognition engine (for example, Google's speech recognition API). For example, the server obtains text data such as "First, we receive the raw materials, then we conduct a quality inspection, then we start the assembly process. Finally, we inspect the products and prepare them for shipping."

[1280] 3. Creating a production process flow

[1281] The server uses OpenAI's generative AI model to analyze the received text data and generate a production process flow diagram. The prompt is "Generate a production process flow diagram based on the following sentence: First, receive the raw materials, then perform quality inspection, then start assembly, and finally inspect and prepare for shipment."

[1282] The AI ​​model outputs a flow diagram of "Acceptance → Quality inspection → Assembly → Inspection → Preparation for shipment."

[1283] 4. Modifying the flow diagram

[1284] If the generated flow diagram contains any deficiencies or errors, the server notifies the user. The user can then provide additional explanations by voice and send them to the server via their terminal. For example, the user can provide additional information such as, "Some parts need to be re-inspected after assembly."

[1285] The server then performs speech recognition on the speech and converts it into text. The additional information is reflected in the flow diagram, generating a revised diagram.

[1286] 5. Identifying issues and presenting solutions

[1287] The server then uses the AI ​​model again to extract issues from the generated production process flow diagram, such as delays in assembly work.

[1288] Then, based on that, they present solutions, such as proposing "prevent delays by promoting automation of assembly work."

[1289] 6. Slide and video generation

[1290] The server uses Python-pptx to compile the generated flow diagrams, issues, and solutions into slides, which can then be output as videos.

[1291] Finally, the product is provided to the user through a terminal.

[1292] Specific examples

[1293] For example, if a user describes, "First, we receive the raw materials, then we perform quality inspection, then we perform assembly work, and finally we inspect the products and prepare them for shipping," the system will operate as follows:

[1294] 1. Speech recognition of user's explanation

[1295] 2. Convert to text data

[1296] 3. Generate a production process flow diagram: "Receipt → Quality inspection → Assembly → Inspection → Shipping preparation"

[1297] 4. Receive additional information from the user and modify the flow diagram as needed.

[1298] 5. Identify issues and present solutions

[1299] 6. Organize the final results in slide format and provide them to the user

[1300] This system allows for efficient and intuitive optimization of production processes simply by using voice input via a smartphone.

[1301] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1302] Step 1:

[1303] The user verbally describes the production process using the smartphone's microphone. For example, "First, we receive the raw materials, then we conduct quality inspections, then we perform assembly work, and finally we inspect the products and prepare them for shipment." The input is voice data.

[1304] Step 2:

[1305] The terminal records the voice input by the user and sends it to the server. The input is voice data, and the output is a data file sent to the server.

[1306] Step 3:

[1307] The server uses a speech recognition engine (for example, Google's speech recognition API) to convert the received voice data into text data. Specifically, the speech recognition model analyzes the voice signal and generates a corresponding text string. The input is the voice data, and the output is the text data: "First, receive the raw materials, then perform quality inspection, then perform assembly work, and finally inspect and prepare for shipment."

[1308] Step 4:

[1309] The server uses a generative AI model (for example, OpenAI's GPT-3) to analyze the obtained text data and generate a production process flow diagram. The prompt is input to the model in the form of "Generate a production process flow diagram based on the following text: First, receive the raw materials, then perform quality inspection, then perform assembly work. Finally, perform inspection and prepare for shipment." The generative AI model analyzes the text and generates a flow diagram of "Receipt → Quality inspection → Assembly → Inspection → Prepare for shipment." The input is text data, and the output is a production process flow diagram.

[1310] Step 5:

[1311] The server displays the generated flow diagram on the terminal, allowing the user to visually confirm it. The input is the production process flow diagram, and the output is the flow diagram displayed on the terminal.

[1312] Step 6:

[1313] If there are any omissions or errors in the flow diagram, the user can provide additional explanations by voice input, for example, "Some parts need to be re-inspected after assembly." The input is voice data.

[1314] Step 7:

[1315] The device records additional audio data and sends it back to the server. The input is the audio data and the output is the data file sent to the server.

[1316] Step 8:

[1317] The server again uses the speech recognition engine to convert the additional voice data into text data. Specifically, the speech recognition model analyzes the voice signal and generates text such as, "Some parts need to be re-inspected after assembly." The input is voice data, and the output is text data.

[1318] Step 9:

[1319] The server modifies the workflow diagram based on the additional text data. For example, it modifies it to the format "Acceptance → Quality Inspection → Assembly → Re-inspection → Inspection → Shipping Preparation." The input is text data, and the output is the modified production process flow diagram.

[1320] Step 10:

[1321] The server extracts issues from the revised flow diagram and proposes solutions using a generative AI model. For example, a delay in assembly work is extracted as an issue, and "automation of assembly work" is proposed as a solution. The input is the revised production process flow diagram, and the output is the issue and solution.

[1322] Step 11:

[1323] The server uses Python-pptx to compile the generated production process flow diagrams, problems, and solutions into slides, which can then be presented to users in a visually easy-to-understand format. The input is text data containing problems and solutions, and the output is presentation slides.

[1324] Step 12:

[1325] Finally, the terminal provides the generated slides to the user, who can visually review them and take action to improve the production process if necessary. The input is the presentation slides, and the output is the visual information provided to the user.

[1326] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1327] This invention is a system that supports business process redesign (BPR) in a conversational format using natural language, and by combining speech recognition technology, text analysis technology, a generative AI model, and an emotion engine, it generates and modifies business flow diagrams, extracts issues, presents solutions, and makes adjustments based on the user's emotions. This system operates mainly on a server, terminals, and users.

[1328] First, the user starts the voice input application and verbally explains the business flow using the device. For example, they might say, "In this department, we first receive the application form, check the contents, and then circulate it to our superiors." The device records this and sends it to the server as voice data.

[1329] Next, the server converts the received voice data into text data using a speech recognition device. Based on this converted text data, an initial workflow diagram is generated. The generated workflow diagram is then displayed on the user's device.

[1330] The server also uses an emotion engine to recognize emotions from the user's voice. For example, if the user is nervous or excited during the explanation, the server can detect the user's emotional state and adjust the generation or modification of the workflow diagram accordingly.

[1331] If the displayed workflow diagram contains any branching points or missing information, the server detects this and notifies the user. The user then provides additional verbal explanation, which is recorded on the device and sent to the server. The server then recognizes the speech again, converts it into text data, and modifies the workflow diagram.

[1332] The server can also use an emotion engine to continue the dialogue depending on the user's emotional state: for example, if the user is feeling stressed, it can provide feedback to promote relaxation.

[1333] The server also uses the problem extraction engine to extract related issues from the generated business flow diagram. It then generates solutions based on these and presents them to the user. For example, specific proposals such as "We propose the introduction of an electronic approval system to speed up approvals."

[1334] Finally, the server creates slides containing the business flow diagram, issues, and solutions, and converts them into video format. The generated slides and videos are then provided to the user via their device. The user can visually check this information and identify areas for improvement and countermeasures for the business process.

[1335] Specific examples

[1336] 1. Voice Input and Recognition

[1337] The user verbally explains the business flow to the terminal, saying, "First, receive the application form, check the contents, and then forward it to your supervisor." The terminal records the voice and sends it to the server. The server uses a voice recognition engine to convert the voice data into text data that says, "First, receive the application form, check the contents, and then forward it to your supervisor."

[1338] 2. Generate a workflow diagram

[1339] The server analyzes the text data and generates a workflow diagram of "Accept → Confirm → Circulate." It then uses an emotion engine to analyze the user's emotional state and makes adjustments according to their emotions. The server then presents the generated workflow diagram to the user.

[1340] 3. Modifying the Workflow Diagram

[1341] The user verbally explains the additional information, saying, "After circulation, once my superior approves, I will pass it on to the next department." The device records the additional voice and sends it to the server. The server recognizes this and modifies the workflow diagram to "Accept → Confirm → Circulate → Approve → Pass on to the next department." The emotion engine also analyzes the user's emotional state and adjusts the feedback when providing additional explanations.

[1342] 4. Presenting the problem and solution

[1343] The server extracts the issue of "approval delays" and generates a solution: "Propose the introduction of an electronic approval system to speed up approvals." The server converts the final business flow diagram, issue, and solution into slides and videos and provides them to the user. The user reviews the provided slides and videos and identifies areas for improvement in the business process and measures to be taken.

[1344] In this way, the present invention provides a system that efficiently and intuitively supports a series of processes for implementing BPR in a conversational manner using natural language, including adjustments based on emotions.

[1345] The processing flow will be explained below.

[1346] The program processing will be specifically explained below by dividing it into steps.

[1347] ---

[1348] Step 1:

[1349] The user starts the voice input application on the device and verbally explains the business flow, for example, "In this department, we first receive the application form, check its contents, and then forward it to the superior."

[1350] Step 2:

[1351] The device records the user's voice and, when the recording is finished, sends the voice data to the server.

[1352] Step 3:

[1353] The server sends the received voice data to a voice recognition engine, which converts the voice data into text data, which is then stored in a database.

[1354] Step 4:

[1355] The server analyzes the stored text data and uses a contextual analysis engine to extract each step of the process.

[1356] Step 5:

[1357] The server automatically generates an initial workflow diagram based on the extracted business steps, and the generated workflow diagram is displayed on the user's device.

[1358] Step 6:

[1359] The server analyzes the business flow diagram to determine whether there are any branching points or missing information. If any branching points or missing information are detected, the server notifies the user.

[1360] Step 7:

[1361] The user verbally provides additional explanations about branching points and missing information through a voice input application, for example, "After circulation, once the supervisor approves, we will pass it on to the next department."

[1362] Step 8:

[1363] The device records additional audio and sends it back to the server.

[1364] Step 9:

[1365] The server recognizes the received additional voice data and converts it into text data.Then, it modifies the workflow diagram based on the received additional information.

[1366] Step 10:

[1367] The server uses an emotion engine to recognize emotions from the user's voice. For example, if the user is nervous or excited, the server detects that emotional state.

[1368] Step 11:

[1369] The server adjusts the generation and modification of the workflow diagram based on the extracted emotional state. For example, if the user is nervous, the server will adjust the feedback to be gentler.

[1370] Step 12:

[1371] The server uses an issue extraction engine to identify potential problems and bottlenecks based on the business flow diagram.

[1372] Step 13:

[1373] The server automatically generates solutions based on the extracted issues, making specific proposals such as "Propose the introduction of an electronic approval system to speed up approvals."

[1374] Step 14:

[1375] The server compiles the final business flow diagram, issues, and solutions into slides, which are then converted into video format.

[1376] Step 15:

[1377] The server transmits the generated slides and videos to the user's terminal.

[1378] Step 16:

[1379] The user reviews the provided slides and videos to identify areas for improvement and measures for business processes.

[1380] ---

[1381] These are the specific processing steps of the program. This process allows users to review and improve their business processes efficiently simply by explaining the workflow in natural language. The emotion engine provides responses and feedback based on the user's emotions, supporting more effective BPR.

[1382] Example 2

[1383] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1384] In conventional business process redesign (BPR) systems, when generating a workflow diagram from voice input, the process proceeds without taking into account the user's emotional state, which can lead to stress and misunderstandings. Another issue is the inefficiency of the problem extraction and solution presentation processes. Furthermore, there is a lack of a way to present the generated information visually in an easy-to-understand manner.

[1385] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a voice recognition means, a means for converting voice data into text data, and a means for recognizing the user's emotions and adjusting the generation and modification of a workflow diagram based on the emotions. This enables flexible responses according to the user's emotional state, and enables efficient generation and modification of a workflow diagram while reducing stress and misunderstandings.

[1386] "Speech recognition means" refers to a device or program for converting voice data into text data.

[1387] "Means for converting voice data into text data" refers to technology that analyzes recorded voice data and converts its contents into text information.

[1388] "Means for analyzing text data and generating business flow diagrams" refers to technology for extracting business procedures and flows from text data and generating a flowchart that visually represents them.

[1389] The "means for displaying the generated workflow diagram" refers to a device or program for displaying the generated workflow diagram on a terminal display or monitor.

[1390] The "means for receiving additional information from the user" is an interface that allows the user to provide further supplementary explanations or corrections to the workflow diagram verbally or by other means.

[1391] The "means for modifying a workflow diagram using additional information" is a technique for updating or modifying an existing workflow diagram based on additional information received from a user.

[1392] "Means for extracting issues related to business flow and proposing solutions" is a technology for finding problems in the generated business flow diagram and proposing specific solutions to them.

[1393] "Means for converting the generated business flow diagram, issues, and solutions into slides and videos" refers to technology for converting the generated business flow diagram, issues, and solutions into presentation slides or video format in order to present them in a visually easy-to-understand manner.

[1394] The "means for providing slides and videos to the user" refers to a technique for visually presenting the generated slides and videos to the user.

[1395] "Means for recognizing a user's emotions and adjusting the generation and modification of a workflow diagram based on those emotions" is a technology that analyzes the user's emotional state from their voice and actions, and adjusts the progress of the generation and modification of a workflow diagram based on those emotions.

[1396] This invention is a system that supports business process redesign (BPR) in a conversational format using natural language. This system combines speech recognition technology, text analysis technology, a generative AI model, and an emotion engine to generate and modify business flow diagrams, identify issues, present solutions, and make adjustments based on the user's emotions. This system operates primarily on a server, terminals, and users.

[1397] First, the user starts the voice input application and verbally explains the business flow using the device. For example, they might say, "In this department, we first receive the application form, check the contents, and then circulate it to our superiors." The device records this and sends it to the server as voice data.

[1398] Next, the server uses a speech recognition tool to convert the received voice data into text data. Based on this converted text data, an initial workflow diagram is generated. Using a generative AI model, the steps and flow of the business are extracted from the text data, and a flowchart that visually represents them is generated. The generated workflow diagram is then displayed on the user's device.

[1399] The server also uses an emotion engine to recognize emotions from the user's voice. For example, if the user is nervous or excited during the explanation, the server can detect that emotional state and adjust the generation or modification of the workflow diagram accordingly.

[1400] If the displayed workflow diagram contains any branching points or missing information, the server detects this and notifies the user. The user then provides additional verbal explanation, which is recorded on the device and sent to the server. The server then recognizes the speech again, converts it into text data, and modifies the workflow diagram.

[1401] The server also uses a problem extraction engine to extract related issues from the generated business flow diagram. Based on this, it generates solutions and presents them to the user. For example, specific proposals such as "We propose the introduction of an electronic approval system to speed up approvals" are included.

[1402] Finally, the server creates slides containing the business flow diagram, issues, and solutions, and converts them into video format. The generated slides and videos are then provided to the user via their device. The user can visually check this information and identify areas for improvement and countermeasures for the business process.

[1403] As a specific example, the following steps can be mentioned.

[1404] 1. Voice Input and Recognition

[1405] The user verbally explains the business flow to the terminal, saying, "First, receive the application form, check the contents, and then forward it to your supervisor." The terminal records the voice and sends it to the server. The server uses a voice recognition engine to convert the voice data into text data that says, "First, receive the application form, check the contents, and then forward it to your supervisor."

[1406] 2. Generate a workflow diagram

[1407] The server analyzes the text data and generates a workflow diagram of "Accept → Confirm → Circulate." It then uses an emotion engine to analyze the user's emotional state and makes adjustments according to their emotions. The server then presents the generated workflow diagram to the user.

[1408] 3. Modifying the Workflow Diagram

[1409] The user verbally explains the additional information, saying, "After circulation, once my superior approves, I will pass it on to the next department." The device records the additional voice and sends it to the server. The server recognizes this and modifies the workflow diagram to "Accept → Confirm → Circulate → Approve → Pass on to the next department." The emotion engine also analyzes the user's emotional state and adjusts the feedback when providing additional explanations.

[1410] 4. Presenting the problem and solution

[1411] The server extracts the issue of "approval delays" and generates a solution: "Propose the introduction of an electronic approval system to speed up approvals." The server converts the final business flow diagram, issue, and solution into slides and videos and provides them to the user. The user reviews the provided slides and videos and identifies areas for improvement in the business process and measures to be taken.

[1412] As described above, the present invention is a system that implements BPR in a conversational format using natural language, and provides efficient and intuitive support, including adjustments based on emotions.

[1413] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1414] Step 1:

[1415] The user starts a voice input application and verbally explains the business flow. For example, "In this department, we first receive the application form, check the contents, and then circulate it to our superiors." The input is voice data, and the output is a recorded voice file. The device records the user's voice and saves the voice data.

[1416] Step 2:

[1417] The device sends the recorded audio file to the server. The input is the recorded audio file, and the output is the audio data transferred to the server. The device converts the audio file to an appropriate format (e.g., WAV format) and sends it to the server via an HTTP request.

[1418] Step 3:

[1419] The server converts the received voice data into text data using a voice recognition engine (e.g., Google Speech-to-Text API). The input is voice data, and the output is text data. The server passes the voice file to the voice recognition engine and obtains the converted text data.

[1420] Step 4:

[1421] The server uses a generative AI model to extract the workflow from the converted text data and generate a workflow diagram. The input is text data, and the output is workflow diagram data. The server analyzes the text and extracts workflow steps such as "accept → confirm → circulate" based on the content.

[1422] Step 5:

[1423] The server uses an emotion engine to analyze the user's emotions. The input is voice data and text data, and the output is emotional state data. The server passes the voice data to the emotion engine for analysis and obtains results such as whether the user is nervous or excited.

[1424] Step 6:

[1425] The server displays the generated workflow diagram on the user's terminal. The input is the workflow diagram data, and the output is a visual display on the terminal. The server sends the workflow diagram to the user's terminal, and the terminal displays it on the screen.

[1426] Step 7:

[1427] The user verbally provides supplementary information to add explanations or make corrections to the business flow diagram. For example, the user might say, "After circulation, once my superior approves, I will pass it on to the next department." The input is additional voice data, and the output is the recorded supplementary voice data.

[1428] Step 8:

[1429] The device re-records the additional audio data and sends it to the server. The input is the additional audio data and the output is the supplemental audio data transferred to the server. The device again converts the audio file into an appropriate format and sends it to the server.

[1430] Step 9:

[1431] The server recognizes the additional voice data and converts it into text data. The input is the additional voice data and the output is the additional text data. The server uses a voice recognition engine to convert the additional explanation into text data.

[1432] Step 10:

[1433] The server modifies the workflow diagram based on the added text data. The input is the additional text data and the existing workflow diagram, and the output is the modified workflow diagram. Based on the additional explanation, the server updates the workflow diagram to "Accept → Confirm → Circulate → Approve → Send to next department."

[1434] Step 11:

[1435] The server uses the problem extraction engine to extract issues from the generated business flow diagram and presents solutions. The input is the modified business flow diagram, and the output is data on the issues and solutions. For example, it extracts the issue of "approval delays" and generates a solution such as "proposing the introduction of an electronic approval system."

[1436] Step 12:

[1437] The server creates slides based on the business flow diagram, issues, and solutions, and converts them into presentation slides and video formats. The input is the business flow diagram, issues, and solution data, and the output is the generated slides and videos. The server uses a slide creation tool to generate visually easy-to-understand slides and videos.

[1438] Step 13:

[1439] The server provides the generated slides and videos to the user's terminal. The input is slide and video data, and the output is a visual display on the user's terminal. The server sends the slides and videos to the user's terminal, and the user visually checks them.

[1440] The above are the processing steps of the program of this system, and summarize the specific operations, inputs, and outputs performed at each step.

[1441] (Application example 2)

[1442] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1443] Traditional business process redesign (BPR) is often highly manual, inefficient, and places a heavy burden on employees, especially those working in the field. It is also difficult to consider employees' emotional states, which can negatively impact stress and work efficiency. This invention aims to solve these problems and provide a system for redesigning business processes more efficiently and intuitively.

[1444] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1445] In this invention, the server includes a speech recognition unit, a means for converting speech data into text data, a means for analyzing the text data and generating a workflow diagram, a means for displaying the generated workflow diagram, a means for receiving additional information from a user, a means for modifying the workflow diagram using the additional information, a means for extracting issues related to the workflow and presenting solutions, a means for converting the generated workflow diagram, issues, and solutions into slides and videos, a means for providing the slides and videos to the user, an emotion recognition unit for analyzing the user's emotions from speech and providing feedback based on the emotional state, and a means for recording the user's voice when the user gives instructions on a business process and generating and modifying a workflow diagram. This allows on-site employees to intuitively generate and modify a workflow diagram simply by verbally explaining the business process, and provides optimal feedback according to the employee's emotional state, thereby improving work efficiency and reducing stress.

[1446] "Speech recognition means" is a technology that acquires and analyzes what a user says orally as digital voice data.

[1447] The "means for converting voice data into text data" refers to a technology for analyzing acquired digital voice data and converting it into corresponding text data.

[1448] The "means for analyzing text data and generating a business flow diagram" is a technology that automatically generates a flow diagram that visualizes the flow of business operations based on converted text data.

[1449] The "means for displaying the generated workflow diagram" is a technique for visually presenting the generated workflow diagram on the user's terminal.

[1450] The "means for receiving additional information from the user" refers to a technique for obtaining supplementary or corrective information provided by the user.

[1451] The "means for modifying a workflow diagram using additional information" is a technique for updating an existing workflow diagram based on additional information received from a user.

[1452] "Means for extracting issues related to business flow and proposing solutions" is a technology that finds problems in the generated business flow diagram and proposes solutions to those problems.

[1453] "Means for converting the generated business flow diagram, issues, and solutions into slides and videos" refers to a technology for converting business flow diagrams, extracted issues, and their solutions into visual slide and video formats.

[1454] The "means for providing slides and videos to users" refers to a technique for displaying or distributing the generated slides and videos to users.

[1455] "Emotion recognition means" is a technology that analyzes emotions from the user's voice and identifies their emotional state.

[1456] "Means for recording the voice of a user giving instructions on a business process and generating or modifying a business flow diagram" refers to a technology that records the user's verbal explanation of a business process and creates or modifies a business flow diagram based on that data.

[1457] This invention is a system that supports business process redesign (BPR) in a conversational format using natural language. By combining speech recognition technology, text analysis technology, a generative AI model, and an emotion engine, the system generates and modifies business flow diagrams, extracts issues, presents solutions, and makes adjustments based on the user's emotions. This system is mainly operated by a server, terminals, and users.

[1458] System configuration

[1459] 1. Voice recognition: The user verbally explains the business flow and the voice is captured as digital data. This is done using a smartphone with a built-in microphone.

[1460] 2. A method for converting voice data to text data: Analyze the voice data and convert it to the corresponding text data. This conversion is performed using the Python speech_recognition library.

[1461] 3. A method for analyzing text data and generating workflow diagrams: A workflow diagram is automatically generated based on the converted text data. A text generation model in Hugging Face's Transformers library is used.

[1462] 4. A means for displaying the generated workflow diagram: The generated workflow diagram is visually displayed on the user's device, using the smartphone screen.

[1463] 5. Means of receiving additional information from the user: A function for obtaining supplementary or corrective information provided by the user.

[1464] 6. A method for modifying a workflow diagram using additional information: Update an existing workflow diagram based on additional information from the user. This also uses speech recognition and text analysis technologies.

[1465] 7. A means to extract issues related to business processes and propose solutions: Identify problems in the generated business process diagram and propose solutions to those problems. This function uses a generative AI model.

[1466] 8. A means of converting generated workflow diagrams, issues, and solutions into slides and videos: A function to convert workflow diagrams and extracted issues and solutions into visual slide and video formats.

[1467] 9. Means of providing slides and videos to users: The ability to display or distribute the generated slides and videos to users.

[1468] 10. Emotion Recognition: Analyzes emotions from the user's voice and identifies their emotional state. This analysis is done using TextBlob and a customized emotion recognition engine, EmotionRecognizer.

[1469] 11. When giving instructions on a business process, a means for recording the voice and generating and modifying a business flow diagram: A user's verbal explanation of a business process is recorded, and a business flow diagram is created and modified based on that data.

[1470] Specific examples

[1471] 1. Voice Input and Recognition

[1472] The user verbally explains the workflow to the terminal, saying, "In this store, first we place products on the shelves, then we check the inventory and take our turn at the cash register." The terminal records the voice and sends it to the server, which uses a speech recognition engine to convert the voice data into text data.

[1473] 2. Generate a workflow diagram

[1474] The server analyzes the text data and generates a workflow diagram of "Shelf placement → Inventory check → Register shift." It then uses an emotion engine to analyze the user's emotional state and makes adjustments according to their emotions. The server then presents the generated workflow diagram to the user.

[1475] 3. Modifying the Workflow Diagram

[1476] The user verbally explains additional information, saying, "After the cashier changes, we will close the register." The device records the additional voice and sends it to the server. The server recognizes this and modifies the workflow diagram to "Shelf arrangement → Inventory check → Cashier change → Close the register." The emotion engine also analyzes the user's emotional state and adjusts the feedback when providing additional explanations.

[1477] 4. Presenting the problem and solution

[1478] The server extracts the issue that "cash register closing work is often delayed" and generates a solution that "proposes the introduction of an automatic cash register closing system." The server converts the final business flow diagram, issue, and solution into slides and videos and provides them to the user. The user reviews the provided slides and videos and understands the areas for improvement and countermeasures for the business process.

[1479] Prompt Sentence Examples

[1480] Current work flow: Place products on shelves, check inventory, and shift to the cash register.

[1481] Please give me some suggestions for improving this workflow. Please tell me specifically what to do, including the solution.

[1482] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1483] Step 1:

[1484] The user verbally explains the workflow, saying, "In this store, first we place products on the shelves, then we check the inventory and take turns at the cash register." The terminal records the voice and saves it as voice data. The input is the user's voice, and the output is voice data.

[1485] Step 2:

[1486] The server converts the voice data received from the terminal into text data using a voice recognition method. Specifically, it uses the Python speech_recognition library to analyze the voice data and generate Japanese text data. The input is voice data and the output is text data.

[1487] Step 3:

[1488] The server analyzes the text data and generates a workflow diagram. Using the text generation model in Hugging Face's Transformers library, a workflow diagram of "Shelf placement → Stock check → Register shift" is automatically generated from the text data. The input is text data, and the output is a workflow diagram.

[1489] Step 4:

[1490] The generated workflow diagram is sent to the user's terminal and displayed. The terminal visually presents the workflow diagram to the user. The input is the workflow diagram, and the output is the workflow diagram displayed on the terminal screen.

[1491] Step 5:

[1492] The user verbally explains changes to the workflow and additional information, such as "After the cashier change, we will perform the cashier closing work." The terminal records the additional voice and sends it to the server. The input is the user's voice, and the output is voice data.

[1493] Step 6:

[1494] The server converts the additional voice data into text data again using a voice recognition means, with the input being voice data and the output being text data.

[1495] Step 7:

[1496] The server reflects the additional text data in the original workflow diagram and modifies the workflow diagram to "Shelf placement → Stock check → Register shift → Cashier closing work." The input is the new text data and the existing workflow diagram, and the output is the modified workflow diagram.

[1497] Step 8:

[1498] The server extracts issues from the workflow diagram and proposes solutions using a generative AI model. For example, it extracts the issue "cash register closing work is often delayed" and proposes "the introduction of an automatic cash register closing system." The input is the workflow diagram, and the output is the issue and solution.

[1499] Step 9:

[1500] The generated workflow diagram, issues, and solutions are converted into slides and videos and sent to the user's device. The device visually presents the slides and videos to the user. The input is the workflow diagram, issues, and solutions, and the output is the slides and videos.

[1501] Step 10:

[1502] The server uses emotion recognition to analyze the emotions in the user's voice and provides feedback based on the user's emotional state. For example, if the user is feeling stressed, it generates feedback to encourage relaxation. The input is the user's voice and the output is feedback.

[1503] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1504] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1505] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1506] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1507] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1508] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1509] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1510] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1511] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1512] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1513] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1514] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1515] In the above embod...

Claims

1. a speech recognition means; means for converting voice data into text data; A means for analyzing text data and generating a workflow diagram; A means for displaying the generated workflow diagram; means for receiving additional information from the user; a means for modifying the workflow diagram with additional information; A means of extracting issues related to business flow and presenting solutions; A means to convert the generated workflow diagrams, issues, and solutions into slides and videos; A system including a means for providing slides and videos to a user.

2. A voice input means is provided, Allows users to verbally input workflows, The system of claim 1 .

3. a speech analysis means for correcting and completing text data; The system recognizes the user's additional explanations and makes corrections to the workflow diagram. The system of claim 1 .

Images