system

Abstract

Provide a system. [Solution] A means for converting the work flow verbally explained by the user into text data using voice recognition technology; A means for analyzing text data to extract steps of a business flow and generate a business flow diagram; A means for converting verbal corrections and supplementary information added by the user into text data using voice recognition technology again and updating the workflow diagram; A means for automatically analyzing and presenting issues and their solutions based on the generated workflow diagram; A means to automatically generate presentation slides based on business flow diagrams, issues, and solutions; A way to create videos based on automatically generated slides, A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Modern local governments and private companies require business process redesign (BPR) to improve and streamline their business processes. However, there is a problem: the in-house development of BPR is not progressing. One factor contributing to this problem is a lack of specialized knowledge and resources for visualizing business flows and identifying issues. Furthermore, traditional BPR tools require advanced technical knowledge, creating a high hurdle for users. This not only delays the adoption of BPR, but also makes it difficult to achieve optimal process improvements. [Means for solving the problem]

[0005] To address the above-mentioned challenges, the present invention provides a system that converts a user's verbal description of a workflow into text data using speech recognition technology, analyzes the text data to extract workflow steps, and generates a workflow diagram. Furthermore, this system also converts any verbal corrections or supplementary information added by the user into text data using speech recognition technology, updating the workflow diagram. Based on the workflow diagram generated in this way, the system automatically analyzes and presents issues and their solutions. Furthermore, by automatically generating presentation slides based on the workflow diagram, issues, and solutions, and creating videos based on those slides, information can be presented in a format that is easy for users to understand. This allows for effective BPR without specialized knowledge and promotes in-house development.

[0006] "User" refers to the entity that uses the system to explain the workflow and check and modify the workflow diagrams, slides, and videos.

[0007] "Terminal" refers to a device that allows a user to input voice and check workflow diagrams, tasks, solutions, slides, and videos.

[0008] "Server" refers to the computer system that performs the main processing of the system, such as speech recognition, text analysis, generation of business flow diagrams, presentation of problems and solutions, and automatic generation of slides and videos.

[0009] "Speech recognition technology" refers to technology that captures a user's verbal explanation as voice data and converts it into text data.

[0010] "Text data" refers to text information converted using voice recognition technology.

[0011] "Business flow steps" refer to the individual tasks or processes that represent the flow of business.

[0012] A "business flow diagram" is a diagram that illustrates the steps of a business flow, and is generated using a BPMN tool or similar.

[0013] "Issues" refer to problems in business processes identified based on the business flow diagram.

[0014] "Solution" refers to a specific method proposed to address a problem.

[0015] "Presentation slides" refers to slide materials used to visually display information such as business flow diagrams, issues, and solutions.

[0016] "Video" refers to dynamic visual information that is animated and narrated based on presentation slides.

[0017] "BPMN tool" refers to software for creating business flow diagrams using Business Process Model and Notation.

[0018] A "voice recognition engine" refers to software or hardware for converting voice data into text data. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] This invention relates to a system that converts a user's verbal description of a workflow into text data using speech recognition technology, analyzes the text data to extract workflow steps, and generates a workflow diagram. This system also has the ability to convert verbal corrections and supplementary information added by the user back into text data using speech recognition technology, updating the workflow diagram. Furthermore, based on the generated workflow diagram, the system automatically analyzes and presents issues and their solutions. It also automatically generates presentation slides based on the workflow diagram, issues, and solutions, and creates videos based on those slides.

[0041] Explaining program processing in natural language

[0042] Voice Recognition

[0043] 1. The user verbally explains the workflow. For example, "I'd like to explain the monthly closing process for the accounting department. First, we collect all transaction data..."

[0044] 2. The device records the user's voice.

[0045] The recording module will be activated and the microphone will capture your voice.

[0046] 3. The device sends the recorded voice data to the voice recognition engine.

[0047] Converts the audio data into the appropriate format and sends the request to the Engine API.

[0048] 4. The voice recognition engine converts the voice data into text data.

[0049] The engine analyzes the audio waveform and generates text output.

[0050] 5. The server receives the text data obtained from the speech recognition engine.

[0051] The API response is received and the text data is stored in the analysis storage.

[0052] Generate a workflow diagram

[0053] 6. The server analyzes the text data obtained from the speech recognition engine.

[0054] The Natural Language Processing (NLP) module analyzes the text and extracts business flow steps.

[0055] 7. The server generates an initial business flow diagram from the extracted steps.

[0056] Create flow diagrams using BPMN (Business Process Model and Notation) tools.

[0057] 8. The server sends the generated business flow diagram to the terminal.

[0058] The flow diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[0059] Completion of missing information

[0060] 9. The user checks the workflow diagram and verbally provides additional information or corrections if necessary. For example, "There's a data check step missing here."

[0061] 10. The device records the audio again and sends it to the speech recognition engine.

[0062] The recording module will be activated again and the microphone will capture the audio.

[0063] The recording data is sent to the speech recognition engine.

[0064] 11. The speech recognition engine converts the speech back into text data, which is then received by the server.

[0065] As before, we analyze the audio waveform and generate text output.

[0066] Receive the API response and store it as additional information in the analytical storage.

[0067] 12. The server updates the business flow diagram based on the information obtained again.

[0068] Incorporate the new steps into an existing flow diagram and regenerate the updated diagram.

[0069] 13. The server sends the updated workflow diagram to the terminal.

[0070] The regenerated diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[0071] Presenting the problem and solution

[0072] 14. The server automatically analyzes common problems and their solutions based on the generated business flow diagram.

[0073] The issue extraction module analyzes the flow diagram and identifies potential issues from data patterns.

[0074] The solution proposal module searches for suitable solutions from a historical database and generates proposals.

[0075] 15. The terminal prompts the user to confirm the presented problem and solution.

[0076] The problem and solution are displayed on the screen so that the user can check it.

[0077] 16. The user reviews the problem and solution and suggests modifications if necessary. For example, "This solution doesn't apply here, so let's think of another way."

[0078] Slide and video generation

[0079] 17. The server automatically generates presentation slides based on the business flow diagram, issues, and solutions.

[0080] The slide generation module embeds business flow diagrams, challenges, and solutions into templates.

[0081] 18. The server creates a video based on the generated slides and sends it to the terminal.

[0082] A video generation module animates the slides and provides narration.

[0083] A video file is generated and sent to the device as an HTTP response.

[0084] 19. The user reviews the final slides and video and downloads or shares them as desired.

[0085] The device displays the slides and video, which the user can review.

[0086] Specific examples

[0087] 1. The user says, "I'd like to explain the monthly closing process for the accounting department. First, we'll collect all the transaction data..."

[0088] 2. The device records the audio and converts it into text using a speech recognition engine.

[0089] 3. The server extracts steps such as "collect transaction data" and generates an initial business flow diagram.

[0090] 4. The server sends the generated diagram to the terminal, where the user can confirm it.

[0091] 5. The user verbally provides additional information, such as, "There's a data check step missing here."

[0092] 6. The device will record the audio again and convert it into text using a speech recognition engine.

[0093] 7. The server generates an updated workflow diagram with the new information added.

[0094] 8. Based on the diagram, the server provides information such as "common issues in data checking" and "solutions," and automatically generates slides.

[0095] 9. Finally, the presentation slides and video are sent to the device for the user to review.

[0096] The above is a detailed description of the embodiment of the present invention, which enables effective BPR without specialized knowledge and promotes in-house development of business processes.

[0097] The processing flow will be explained below.

[0098] Step 1:

[0099] The user verbally explains the workflow. For example, "I will explain the monthly closing process for the accounting department. First, we collect all transaction data..."

[0100] Step 2:

[0101] The terminal records the user's voice.

[0102] The recording module will be activated and the microphone will capture your voice.

[0103] Step 3:

[0104] The device sends the recorded voice data to a voice recognition engine.

[0105] Converts the audio data into the appropriate format and sends the request to the Engine API.

[0106] Step 4:

[0107] A voice recognition engine converts the voice data into text data.

[0108] The engine analyzes the audio waveform and generates text output.

[0109] Step 5:

[0110] The server receives the text data obtained from the speech recognition engine.

[0111] The API response is received and the text data is stored in the analysis storage.

[0112] Step 6:

[0113] The server analyzes the text data and extracts the steps of the business flow.

[0114] The Natural Language Processing (NLP) module analyzes the text and extracts business flow steps.

[0115] Step 7:

[0116] The server generates an initial business flow diagram from the extracted steps.

[0117] Create flow diagrams using BPMN (Business Process Model and Notation) tools.

[0118] Step 8:

[0119] The server sends the generated business flow diagram to the terminal.

[0120] The flow diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[0121] Step 9:

[0122] The user checks the workflow diagram and verbally provides additional information or corrections if necessary. For example, "There's a data check step missing here."

[0123] Step 10:

[0124] The device will record the audio again and send it to the speech recognition engine.

[0125] The recording module will be activated again and the microphone will capture the audio.

[0126] The recording data is sent to the speech recognition engine.

[0127] Step 11:

[0128] The voice recognition engine converts the voice data into text data again, which is received by the server.

[0129] As before, we analyze the audio waveform and generate text output.

[0130] Receive the API response and store it as additional information in the analytical storage.

[0131] Step 12:

[0132] The server updates the business flow diagram based on the information retrieved again.

[0133] Incorporate the new steps into an existing flow diagram and regenerate the updated diagram.

[0134] Step 13:

[0135] The server sends the updated business process diagram to the terminal.

[0136] The regenerated diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[0137] Step 14:

[0138] The server automatically analyzes common problems and their solutions based on the generated business flow diagram.

[0139] The issue extraction module analyzes the flow diagram and identifies potential issues from data patterns.

[0140] The solution proposal module searches for suitable solutions from a historical database and generates proposals.

[0141] Step 15:

[0142] The terminal allows the user to confirm the presented problem and solution.

[0143] The problem and solution are displayed on the screen so that the user can check it.

[0144] Step 16:

[0145] The user reviews the problem and solution and suggests modifications if necessary, for example, "This solution doesn't apply here, so let's think of another way."

[0146] Step 17:

[0147] The server automatically generates presentation slides based on the business flow diagram, issues, and solutions.

[0148] The slide generation module embeds business flow diagrams, challenges, and solutions into templates.

[0149] Step 18:

[0150] The server creates a video based on the generated slides and sends it to the terminal.

[0151] A video generation module animates the slides and provides narration.

[0152] A video file is generated and sent to the device as an HTTP response.

[0153] Step 19:

[0154] Users can review the final slides and videos and download or share them as desired.

[0155] The device presents slides and videos, which the user reviews.

[0156] The above is a detailed description of the specific processing steps of the program.

[0157] Example 1

[0158] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0159] Traditionally, the process of formulating workflows was largely manual, requiring time and effort. Furthermore, reviewing and improving workflows required extensive specialized knowledge and skills, making the process inefficient. Furthermore, updating workflow diagrams, identifying issues, and proposing solutions were complex, creating a need for automation.

[0160] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0161] In this invention, the server includes means for converting a business process described orally by a user into text data using speech recognition technology, means for analyzing the text data to extract business process steps and generate a business process diagram, means for converting verbal corrections and supplementary information added by the user into text data again using speech recognition technology and updating the business process diagram, means for automatically analyzing and presenting issues and their solutions based on the generated business process diagram, means for automatically generating presentation slides based on the business process diagram, issues, and solutions, and means for creating videos based on the automatically generated slides. This enables effective improvement of business processes without specialized knowledge and promotes in-house development of business processes.

[0162] A "user" is someone who uses the system to explain the workflow and add corrections or supplementary information.

[0163] "Speech recognition technology" is a technology that analyzes the voice data of a user's oral explanation and converts it into text data.

[0164] "Text data" refers to the text information of voice data converted using voice recognition technology.

[0165] "Analysis" refers to the process of classifying text data, interpreting its meaning and structure, and extracting business flow steps.

[0166] A "business flow step" refers to an individual task or processing stage in a business process.

[0167] A "business flow diagram" is a diagram that visually represents the steps of a business flow.

[0168] An "issue" is a problem or obstacle that occurs in a business flow.

[0169] A "solution" is a method or means for solving a particular problem.

[0170] "Presentation slides" are materials used to visually present information such as business flow diagrams, issues, and solutions.

[0171] "Video" refers to dynamic video content created by adding animation and narration to automatically generated slides.

[0172] This invention relates to a system that uses speech recognition technology to generate a workflow diagram based on a user's verbal description of the workflow, thereby improving the efficiency of business processes. Specifically, the system converts the speech data into text data, analyzes it, extracts the workflow steps, and automatically generates a workflow diagram. Furthermore, the system is characterized by the fact that the workflow diagram is constantly updated with the latest information as the user verbally provides additional information or corrections, and automatically presents issues and their solutions.

[0173] To implement this system, the following hardware and software are required:

[0174] Device: A device used to record audio and communicate with the server. Examples include smartphones, tablets, and computers.

[0175] Server: The central computer system that provides the speech recognition and text analysis technologies. This may include a cloud server or a dedicated server.

[0176] Speech recognition engine: Software that converts a user's voice into text data. Typical engines include the Google® Cloud Speech-to-Text API and the IBM Watson® Speech to Text API.

[0177] Natural Language Processing (NLP) module: A software module for analyzing text data and extracting business flow steps.

[0178] BPMN (Business Process Model and Notation) tool: A dedicated tool for generating business flow diagrams.

[0179] Data analysis module: Software for automatically analyzing issues and their solutions from business flow diagrams.

[0180] Slide generation module: A module for automatically generating slides for presentations.

[0181] Video Generation Module: A module for creating videos based on generated slides.

[0182] The specific operating procedure of the system is as follows: first, the user verbally explains the workflow, which is then recorded by the device. The recorded voice data is sent to the server, where it is converted into text data by a speech recognition engine. Next, the server analyzes the text data using an NLP module and extracts the workflow steps. A BPMN tool generates a workflow diagram based on the extracted steps.

[0183] If the user verbally adds or modifies information to the diagram, the device records the voice again and sends it to a speech recognition engine to convert it into text data. The server updates the workflow diagram based on the new information and automatically analyzes and presents the issues and solutions. The server then uses a slide generation module to create presentation slides based on the workflow diagram, issues, and solutions. Furthermore, the server uses a video generation module to create a video based on the created slides and sends it to the device.

[0184] Specific examples

[0185] The user verbally explains, "I'd like to explain the monthly closing process for the accounting department. First, we'll collect all transaction data..." The device records the speech and converts it into text data using a voice recognition engine. The server extracts steps such as "collect transaction data" and generates an initial business flow diagram. The server sends the diagram it has generated to the device, where the user can confirm it.

[0186] The user verbally provides additional information, saying, "There's a data check step missing here." The device records the voice again and converts it into text data using a speech recognition engine. The server adds the new information and generates an updated business flow diagram. Based on the diagram, the server provides "common issues in data checks" and "solutions," and automatically generates slides. Finally, the presentation slides and video are sent to the device, where the user can review them.

[0187] The above is a detailed description of the embodiment of the present invention. This system enables effective improvement of business processes without specialized knowledge, and promotes automation and in-house production of business flows.

[0188] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0189] Step 1:

[0190] The user verbally explains the workflow. For example, "I'll explain the monthly closing process for the accounting department. First, we'll collect all transaction data..." This explanation becomes the initial data input.

[0191] Step 2:

[0192] The terminal records the user's voice. The recording module is activated and the device's microphone captures the voice. This captured voice data becomes the input data for the next step.

[0193] Step 3:

[0194] The device sends the recorded voice data to the speech recognition engine. The voice data is converted to an appropriate format (e.g., WAV, FLAC) and sent to the speech recognition engine using an HTTP request. This makes the voice data the target data for conversion.

[0195] Step 4:

[0196] The speech recognition engine converts the speech data into text data. It analyzes the speech waveform and generates text output. For example, the text data generated is "Collecting transaction data." This text data becomes the input data for analysis.

[0197] Step 5:

[0198] The server receives the text data obtained from the speech recognition engine. The JSON-formatted text data is received as an API response and stored in the analysis storage. This stored data becomes the input data for subsequent analysis processing.

[0199] Step 6:

[0200] The server analyzes the text data obtained from the speech recognition engine. The NLP module analyzes the text data, extracts nouns and verbs, and identifies the steps in the business flow. For example, steps such as "collect transaction data" and "verify data" are extracted. The results of this analysis become the input data for generating the flow diagram.

[0201] Step 7:

[0202] The server extracts the business flow steps from the analysis results. The NLP module lists the steps obtained and clarifies the order and relationships required to generate a business flow diagram. This becomes the preparatory data for generating the flow diagram.

[0203] Step 8:

[0204] The server generates a business flow diagram based on the extracted steps. Using a BPMN tool, a business flow diagram is created that visually represents a series of steps. For example, a flow diagram of "Collect transaction data" → "Confirm data" → "Complete monthly closing process" is generated. This generated flow diagram becomes the input data for the next import.

[0205] Step 9:

[0206] The server sends the generated business flow diagram to the terminal. The flow diagram is converted into an image or XML format and sent to the terminal as an HTTP response. This allows the user to check the specific flow diagram.

[0207] Step 10:

[0208] The user checks the workflow diagram and verbally provides additional information or corrections if necessary. For example, they might say, "There's a data check step missing here." This additional information becomes the input data for the next step.

[0209] Step 11:

[0210] The device records the audio again and sends it to the speech recognition engine. The recording module is then activated again, capturing additional audio, converting it to the appropriate format, and sending it to the engine, which then uses it as input data for reconversion.

[0211] Step 12:

[0212] The speech recognition engine converts the speech into text data again. It analyzes the additional speech and generates text containing "data check." This text data becomes the input data for analysis.

[0213] Step 13:

[0214] The server receives and analyzes the new text data. It analyzes the updated text data and recognizes the new step "Data Check." This updated data becomes the updated input data for the flow diagram.

[0215] Step 14:

[0216] The server generates an updated business flow diagram. The new steps are incorporated into the existing flow diagram, and the flow diagram is regenerated in the format of "Collect transaction data" → "Data check" → "Data confirmation" → "Monthly closing process completed." This flow diagram will be the updated data.

[0217] Step 15:

[0218] The server sends the updated business flow diagram to the terminal. The regenerated flow diagram is converted back to PNG or SVG format and sent to the terminal. The user can then review it.

[0219] Step 16:

[0220] The server analyzes the issues and solutions based on the workflow diagram. The data analysis module analyzes the workflow diagram and identifies issues at specific steps. For example, an issue such as "data checking takes a long time" is extracted. This issue and solution become the input data for the next generated slide.

[0221] Step 17:

[0222] The terminal presents the problem and the solution to the user. The problem and the solution received from the server are displayed on the screen so that the user can confirm them. This allows the user to decide on the next action.

[0223] Step 18:

[0224] The user checks the problem and solution and gives instructions for corrections as necessary. For example, the user may give verbal instructions such as, "It would be better to use the current tool, so please think of another solution." This additional instruction becomes prompt data for the next step.

[0225] Step 19:

[0226] The server generates presentation slides based on the business flow diagram, issues, and solutions. The slide generation module embeds the business flow diagram, issues, and solutions into a template and generates a slide-format file. The output is a PowerPoint or PDF.

[0227] Step 20:

[0228] The server generates a video based on the slides and sends it to the device. The video generation module animates the generated slides, synthesizes the narration from the text, and creates a video file. This video becomes the final output data for the user.

[0229] Step 21:

[0230] The user reviews the final slides and video, and downloads or shares them as needed. The device displays the slides and video, allowing the user to review them and proceed to the next step in the process.

[0231] (Application example 1)

[0232] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0233] Optimizing business processes in factories is important from the perspective of improving efficiency and productivity on-site, but conventional methods require specialized knowledge and require a great deal of time and effort. In particular, it has been difficult to grasp the real-time progress of work within a factory and immediately present optimization proposals. In addition, organizing business flows and creating presentation materials also require time and effort, which leads to a decline in work efficiency. For this reason, there is a need for a system that can efficiently manage business processes within a factory and make immediate proposals.

[0234] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0235] In this invention, the server includes: means for converting a work flow verbally explained by a user into text data using voice recognition technology; means for analyzing the text data to extract workflow steps and generate a workflow diagram; means for converting verbal corrections and supplementary information added by the user into text data using voice recognition technology again to update the workflow diagram; means for automatically analyzing and presenting problems and solutions based on the generated workflow diagram; means for automatically generating presentation images based on the workflow diagram, problems, and solutions; means for creating videos based on the automatically generated images; means for using a robot equipped with a voice recognition module that allows a user to verbally explain factory operations and work instructions in real time; and means for analyzing voice data recorded by the robot and providing optimization suggestions based on the workflow diagram. This enables efficient management of factory business processes and real-time optimization suggestions on-site.

[0236] "User" refers to a person who uses the system to verbally explain the workflow and provide corrections and supplementary information.

[0237] "Verbal explanation" refers to the act of using voice to communicate the flow of work and correction information.

[0238] "Voice recognition technology" refers to the technology that analyzes voice and converts it into text data.

[0239] "Text data" refers to text data converted from speech using speech recognition technology.

[0240] "Workflow" refers to a series of steps or procedures for accomplishing a particular task or work.

[0241] A "step" refers to an individual unit of work within a workflow.

[0242] A "workflow diagram" refers to a visual representation of a workflow.

[0243] A "problem" refers to a glitch or inefficiency that occurs within a workflow.

[0244] A "solution" refers to a specific method or proposal for solving a problem.

[0245] "Presentation images" refers to diagrams and presentation slides that visually illustrate workflows, business problems, and solutions.

[0246] "Video creation" refers to the process of creating visual explanations in video format based on presentation images.

[0247] A "robot equipped with a voice recognition module" refers to a robot that has the ability to record voice data and analyze the voice in real time.

[0248] "Optimization proposals based on workflow diagrams" refers to proposals that analyze workflow diagrams to identify problems and show how to improve them.

[0249] "Real-time" refers to processing and providing information immediately, without delay.

[0250] This invention is a system for efficiently managing work processes in factories and making optimization proposals in real time. This system uses voice recognition technology to convert a user's verbal explanation into text data, analyzes that data to generate a workflow diagram, and finally creates images and videos for presentations based on the proposals.

[0251] System Overview

[0252] The system includes the following elements:

[0253] 1. Robot equipped with a voice recognition module

[0254] It records factory operations and work instructions in real time and converts user voice into text data.

[0255] Specific hardware: robot, microphone

[0256] Software used: Google Speech-to-Text API

[0257] 2. Server

[0258] Analyzes character data and extracts workflow steps.

[0259] Generate workflow diagrams to analyze and present problems and their solutions.

[0260] Automatically generate images and videos for presentations.

[0261] Software used: nltk, pydotplus, PaddleOCR

[0262] System action

[0263] Voice Recognition

[0264] The robot embedded in the device records work instructions and explanations in real time within the factory and converts them into text data using the Google Speech-to-Text API, which the device then sends to a server.

[0265] Text data analysis

[0266] The server receives the text data and uses natural language processing tools such as nltk to extract each step of the workflow. Pydotplus is used to generate a workflow diagram from the extracted steps.

[0267] Presenting the problem and solution

[0268] Based on the generated workflow diagram, the server automatically analyzes potential problems, searches for appropriate solutions from past data, and presents them to the user. This analysis utilizes data pattern mining technology.

[0269] Image and video generation for presentations

[0270] The server generates presentation images and explanatory videos based on the generated workflow diagram and analysis results. The presentation images are generated by embedding information in templates, and the explanatory videos are generated by animating the images and adding narration.

[0271] Specific examples

[0272] 1. The user says, "I will explain the assembly procedure for the production line. First, prepare all the parts, then connect part A to part B. After that, perform a quality check..."

[0273] 2. The device records the audio and converts it into text using the Google Speech-to-Text API.

[0274] 3. The server receives the text data and extracts steps such as "preparing parts," "connecting parts A and B," and "quality check."

[0275] 4. The server generates a workflow diagram based on the extracted information and displays it on the terminal for the user to review.

[0276] 5. The user verbally adds additional information, saying, "After the quality check, there will be packaging."

[0277] 6. The device records the audio again and converts it into text data using the Google Speech-to-Text API.

[0278] 7. The server takes in the new information and generates an updated workflow diagram.

[0279] 8. Based on this workflow diagram and analysis results, the server provides optimization suggestions and generates presentation images and explanatory videos.

[0280] Prompt Sentence Examples

[0281] "Please explain in detail the workflow of your factory's production line. Then, please add any missing steps or areas for improvement. Also, please add details of each process."

[0282] This system allows users to effectively optimize factory operations without specialized knowledge, making it possible to improve the efficiency of work flows.

[0283] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0284] Step 1:

[0285] The terminal records the work instructions and workflows that users verbally explain in the factory. Specifically, a microphone installed on the robot captures the user's voice, and the recording module converts the voice into a digital format. The recording data is generated and sent to the server. The output based on the input is digital voice data.

[0286] Step 2:

[0287] The server sends the received digital voice data to the Google Speech-to-Text API, which converts the voice into text data (character data). The server then calls the API of the speech recognition engine and converts the voice input into text format. The input data is voice data, and the output data is character data.

[0288] Step 3:

[0289] The server analyzes the text data and extracts each step of the workflow. Specifically, the Natural Language Processing (NLP) module analyzes the text data and identifies a series of work steps from the content. The input is the text data, and the output is a list of the analyzed work steps.

[0290] Step 4:

[0291] The server generates a workflow diagram based on the extracted work steps. The server uses pydotplus to create a flow diagram that connects each step. The input is a list of work steps, and the output is a workflow diagram. The flow diagram is generated in image or XML format and sent to the terminal.

[0292] Step 5:

[0293] The user checks the workflow diagram and provides additional verbal explanations if there is missing information or if corrections are required. Specifically, when the user explains again, the device re-records the voice and sends it to the server. The input data is the user's voice, and the output data is voice data including supplementary information.

[0294] Step 6:

[0295] The server again converts the audio data of the additional explanation into text data using the Google Speech-to-Text API and analyzes the text data as before. The NLP module analyzes the added text information and updates the existing workflow diagram. The input is the audio data containing the supplementary information, and the output is the updated text data.

[0296] Step 7:

[0297] The server regenerates the workflow diagram based on the updated work steps. It uses pydotplus to incorporate the new steps into the existing diagram and recreates the updated diagram. The input is the updated list of work steps, and the output is the updated workflow diagram. The updated diagram is also sent to the terminal.

[0298] Step 8:

[0299] The server automatically analyzes and presents potential problems and their solutions based on the workflow diagram. Using a problem extraction module and a solution presentation module, the server analyzes the generated workflow diagram, identifies problems, and searches a database for appropriate solutions. The input is the workflow diagram, and the output is a list of problems and solutions, which is also sent to the terminal.

[0300] Step 9:

[0301] The server automatically generates images and videos for presentations based on the workflow diagram, problems, and solutions. The server uses a slide generation module and a video generation module to embed information in templates and create slides and videos. The input is the workflow diagram, problems, and a list of solutions, and the output is the images and video files for presentations. Finally, these are also sent to the terminal for the user to review.

[0302] Each step is explained in detail to help you understand how the system streamlines factory operations in real time.

[0303] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0304] This invention relates to a system that converts a user's verbal description of a workflow into text data using speech recognition technology, analyzes the text data to extract workflow steps, and generates a workflow diagram. This system also has the ability to convert verbal corrections and supplementary information added by the user back into text data using speech recognition technology, updating the workflow diagram. Based on the generated workflow diagram, the system automatically analyzes and presents issues and their solutions. Furthermore, the system automatically generates presentation slides based on the workflow diagram, issues, and solutions, and creates videos based on those slides.

[0305] Furthermore, by incorporating an emotion engine that recognizes emotions from the user's voice, the present invention realizes adaptive suggestions and information presentation based on the user's emotional state. This emotion engine also simultaneously acquires and stores user emotion data, which can be used for later analysis and improvement.

[0306] Explaining program processing in natural language

[0307] Speech and Emotion Recognition

[0308] 1. The user verbally explains the workflow. For example, "I'll explain the monthly closing process for the accounting department. First, we collect all transaction data..."

[0309] 2. The device records the user's voice.

[0310] The recording module will be activated and the microphone will capture your voice.

[0311] 3. The device sends the recorded voice data to the speech recognition engine and emotion engine.

[0312] Converts the audio data into the appropriate format and sends the request to the Engine API.

[0313] 4. The speech recognition engine converts the voice data into text data.

[0314] The engine analyzes the audio waveform and generates text output.

[0315] 5. The emotion engine recognizes the user's emotions from the voice data.

[0316] The emotion analysis module analyzes the speech features and outputs the emotional state.

[0317] 6. The server receives the text data and emotion data obtained from the speech recognition engine and emotion engine.

[0318] The API response is received and the text data and emotion data are stored in the storage for analysis.

[0319] Business flow diagram generation and emotional adaptation

[0320] 6. The server analyzes the text data and extracts the steps of the business flow.

[0321] The Natural Language Processing (NLP) module analyzes the text and extracts business flow steps.

[0322] 7. The server generates an initial business flow diagram from the extracted steps.

[0323] Create flow diagrams using BPMN (Business Process Model and Notation) tools.

[0324] 8. The server sends the generated business flow diagram to the terminal.

[0325] The flow diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[0326] Completion of missing information and emotional adaptation

[0327] 9. The user checks the workflow diagram and verbally provides additional information or corrections if necessary. For example, "There's a data check step missing here."

[0328] 10. The device records the audio again and sends it to the speech recognition engine and emotion engine.

[0329] The recording module will be activated again and the microphone will capture the audio.

[0330] The recording data is sent to the speech recognition engine and emotion engine.

[0331] 11. The voice recognition engine converts the voice data back into text data, which is received by the server.

[0332] As before, we analyze the audio waveform and generate text output.

[0333] Receive the API response and store it as additional information in the analytical storage.

[0334] 12. The emotion engine again recognizes the user's emotion from the voice data, and the server receives it.

[0335] The emotion analysis module analyzes the speech features and outputs the emotional state.

[0336] The API response is received and the added emotion data is also stored in the storage for analysis.

[0337] 13. The server updates the workflow diagram based on the information and emotion data obtained again.

[0338] New steps are incorporated into existing flow diagrams, and the diagrams are adaptively regenerated based on emotion data.

[0339] 14. The server sends the updated business flow diagram to the terminal.

[0340] The regenerated diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[0341] Presenting the problem and solution

[0342] 15. The server automatically analyzes common problems and their solutions based on the generated business flow diagram.

[0343] The issue extraction module analyzes the flow diagram and identifies potential issues from data patterns.

[0344] The solution suggestion module also takes into account the emotional data and suggests the best solution.

[0345] 16. The device prompts the user to confirm the presented problem and solution.

[0346] The problem and solution are presented on the screen, allowing the user to see it in the way that is most receptive to their emotions.

[0347] 17. The user reviews the problem and solution and suggests modifications if necessary. For example, "This solution doesn't work here, so let's think of another way."

[0348] Slide and video generation

[0349] 18. The server automatically generates presentation slides based on the business flow diagram, issues, and solutions.

[0350] The slide generation module embeds workflow diagrams, challenges, and solutions into templates, applying emotionally sensitive layouts and designs.

[0351] 19. The server creates a video based on the generated slides and sends it to the terminal.

[0352] A video generation module animates the slides and provides narration.

[0353] Adjust the tone and speed of narration based on emotional data.

[0354] A video file is generated and sent to the device as an HTTP response.

[0355] 20. User reviews the final slides and video and downloads or shares as desired.

[0356] The device displays slides and videos, which the user reviews.

[0357] Specific examples

[0358] 1. The user says, "I'd like to explain the monthly closing process for the accounting department. First, we'll collect all the transaction data..."

[0359] 2. The device records the audio and analyzes it using a voice recognition engine and emotion engine.

[0360] 3. The server extracts steps such as "collect transaction data" and generates an initial workflow diagram. If the user's emotion is recognized as "excited," the information is presented in a calm tone.

[0361] 4. The server sends the generated diagram to the terminal, where the user can confirm it.

[0362] 5. The user verbally provides additional information, such as, "There's a data check step missing here."

[0363] 6. The device records the audio again and analyzes it again using the speech recognition engine and emotion engine.

[0364] 7. The server generates an updated workflow diagram by adding new information and presents it adaptively based on the emotion data.

[0365] 8. The server automatically generates slides based on the diagram, providing information such as "common issues in data checking" and "solutions." The slide design is adjusted based on the emotion.

[0366] 9. Finally, the presentation slides and video are sent to the device for the user to review.

[0367] The above is a detailed description of the embodiment of this invention. This makes it possible to effectively implement BPR without specialized knowledge and promote in-house development of business processes. Furthermore, by combining it with an emotion engine, the user experience is improved, and more effective proposals and information presentation are realized.

[0368] The processing flow will be explained below.

[0369] Step 1:

[0370] The user verbally explains the workflow. For example, "I will explain the monthly closing process for the accounting department. First, we collect all transaction data..."

[0371] Step 2:

[0372] The terminal records the user's voice.

[0373] The recording module will be activated and the microphone will capture your voice.

[0374] Step 3:

[0375] The terminal transmits the recorded voice data to a voice recognition engine and an emotion engine.

[0376] Converts the recording into the appropriate format and sends the request to the engine API.

[0377] Step 4:

[0378] A voice recognition engine converts the voice data into text data.

[0379] The engine analyzes the audio waveform and generates text output.

[0380] Step 5:

[0381] The emotion engine recognizes the user's emotion from the voice data.

[0382] The emotion analysis module analyzes the speech features and outputs the emotional state.

[0383] Step 6:

[0384] A server receives the text data and emotion data obtained from the speech recognition engine and emotion engine.

[0385] The API response is received and the text data and emotion data are stored in the storage for analysis.

[0386] Step 7:

[0387] The server analyzes the text data and extracts the steps of the business flow.

[0388] The Natural Language Processing (NLP) module analyzes the text and extracts business flow steps.

[0389] Step 8:

[0390] The server generates an initial business flow diagram from the extracted steps.

[0391] Create flow diagrams using BPMN (Business Process Model and Notation) tools.

[0392] Step 9:

[0393] The server sends the generated business flow diagram to the terminal.

[0394] The flow diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[0395] Step 10:

[0396] The user checks the workflow diagram and verbally provides additional information or corrections if necessary.

[0397] Step 11:

[0398] The device records the audio again and sends it to the speech recognition engine and emotion engine.

[0399] The recording module will be activated again and the microphone will capture the audio.

[0400] The recording data is sent to the speech recognition engine and emotion engine.

[0401] Step 12:

[0402] The voice recognition engine converts the voice data into text data again.

[0403] The engine analyzes the audio waveform and generates text output.

[0404] Step 13:

[0405] The emotion engine again recognizes the user's emotion from the voice data.

[0406] The emotion analysis module analyzes the speech features and outputs the emotional state.

[0407] Step 14:

[0408] The server receives the reacquired text data and emotion data.

[0409] The API response is received and the text data and emotion data are stored in the storage for analysis.

[0410] Step 15:

[0411] The server updates the workflow diagram based on the information and emotion data acquired again.

[0412] New steps are incorporated into existing flow diagrams, and the diagrams are adaptively regenerated based on emotion data.

[0413] Step 16:

[0414] The server sends the updated business process diagram to the terminal.

[0415] The regenerated diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[0416] Step 17:

[0417] The server automatically analyzes common problems and their solutions based on the generated business flow diagram.

[0418] The issue extraction module analyzes the flow diagram and identifies potential issues from data patterns.

[0419] The solution suggestion module also takes into account the emotional data and suggests the best solution.

[0420] Step 18:

[0421] The terminal allows the user to confirm the presented problem and solution.

[0422] The problem and solution are presented on the screen, allowing the user to see it in the way that is most receptive to their emotions.

[0423] Step 19:

[0424] The user checks the issues and solutions and instructs corrections as necessary.

[0425] Step 20:

[0426] The server automatically generates presentation slides based on the business flow diagram, issues, and solutions.

[0427] The slide generation module embeds workflow diagrams, challenges, and solutions into templates, applying emotionally sensitive layouts and designs.

[0428] Step 21:

[0429] The server creates a video based on the generated slides and sends it to the terminal.

[0430] A video generation module animates the slides and provides narration.

[0431] Adjust the tone and speed of narration based on emotional data.

[0432] A video file is generated and sent to the device as an HTTP response.

[0433] Step 22:

[0434] Users can review the final slides and videos and download or share them as desired.

[0435] The above is a detailed description of the embodiment of this invention. This makes it possible to effectively implement BPR without specialized knowledge and promote in-house development of business processes. Furthermore, by combining it with an emotion engine, the user experience is improved, and more effective proposals and information presentation are realized.

[0436] Example 2

[0437] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0438] Conventional business process management systems make it difficult for users to create and update business flows concisely and quickly. Furthermore, they are unable to propose process improvements that take into account user emotions and immediate feedback, limiting the actual improvement of business efficiency. Furthermore, creating presentation materials and videos requires a great deal of effort and time, which means that insufficient resources can be allocated to essential business improvements.

[0439] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0440] In this invention, the server includes means for converting a workflow described orally by a user into text data using speech recognition technology, means for analyzing the text data to extract workflow steps and generate a workflow diagram, means for converting verbal corrections and supplementary information added by the user into text data using speech recognition technology again and updating the workflow diagram, means for automatically analyzing and presenting problems and their solutions based on the generated workflow diagram, means for automatically generating presentation slides based on the workflow diagram, problems, and solutions, means for creating videos based on the automatically generated slides, means for recording the user's explanation, means for analyzing and recording emotion data, and means for processing the user's voice data using a speech recognition engine and an emotion engine.This enables users to not only quickly and easily create and update workflows, but also enables adaptive suggestions based on emotion data and efficient creation of presentation materials and videos.

[0441] "Voice recognition technology" is a technology that analyzes a user's voice and converts it into text data.

[0442] "Text data" is a data format in which the user's voice is converted into text information.

[0443] A "business flow" is a visual representation of each step in a business process.

[0444] A "business flow diagram" is a graphical representation of the steps of a business flow extracted from text data.

[0445] "Emotion analysis" is a technology that recognizes and analyzes a user's emotional state from their voice data.

[0446] "Emotion data" is data that indicates the emotional state of the user obtained by emotion analysis.

[0447] "Presentation slides" are materials used to visually present business flow diagrams, challenges, and solutions.

[0448] "Video" refers to video content in the form of animated presentation slides with narration.

[0449] A "voice recognition engine" is a software module that analyzes voice waveforms and converts them into text data.

[0450] The "emotion engine" is a software module that analyzes the features of the user's voice data and recognizes their emotional state.

[0451] A "server" is a computer system that performs processes such as extracting steps in a business flow, generating business flow diagrams, analyzing and presenting problems and solutions, and generating slides and videos.

[0452] A "terminal" is a device that allows a user to input voice and check the generated business flow diagram, issues, and solutions.

[0453] This system converts a user's verbal description of a workflow into text data using speech recognition technology, analyzes the text data to extract workflow steps, and generates a workflow diagram. This system also has the ability to convert verbal corrections and supplementary information added by the user back into text data using speech recognition technology, updating the workflow diagram. Furthermore, it automatically analyzes and presents issues and solutions based on the generated workflow diagram. It also automatically generates presentation slides based on the workflow diagram, issues, and solutions, and creates videos based on those slides. Additionally, by incorporating an emotion engine that recognizes emotions from the user's voice, the system can adaptively present suggestions and information based on the user's emotional state.

[0454] Hardware and software used

[0455] This system uses the following hardware and software:

[0456] 1. Hardware:

[0457] Device: The device where the user inputs voice (e.g., smartphone, tablet, computer).

[0458] Microphone: An audio input device for capturing the user's voice.

[0459] Server: A computer system that processes and analyzes data.

[0460] 2. Software:

[0461] Speech recognition engine: A technology that converts speech waveforms into text, such as the Google Cloud Speech-to-Text API.

[0462] Emotion engine: Technology that analyzes emotions from voice data, such as IBM Watson Tone Analyzer.

[0463] NLP module: Natural language processing technology that analyzes text data and extracts business flow steps.

[0464] BPMN tool: A Business Process Model and Notation tool for generating business flow diagrams.

[0465] Slide generation module: Software that automatically generates slides for presentations, such as the Google Slides API.

[0466] Video Generation Module: Software that generates videos from slides, such as Adobe Premiere Pro API.

[0467] Specific examples of program processing

[0468] 1. The user verbally explains the workflow. For example, "I'll explain the monthly closing process for the accounting department. First, we collect all transaction data..."

[0469] 2. The device records the audio.

[0470] The recording module will be activated and the microphone will capture the audio.

[0471] 3. The device sends the recorded voice data to the speech recognition engine and emotion engine.

[0472] Converts the voice data into the appropriate format and sends requests to the speech recognition engine and emotion engine APIs.

[0473] 4. The speech recognition engine converts the voice data into text data.

[0474] Analyzes audio waveforms and generates text output.

[0475] 5. The emotion engine recognizes the user's emotions from the voice data.

[0476] Analyzes speech features and outputs emotional state.

[0477] 6. The server receives the text data and emotion data and stores them in the analysis storage.

[0478] 7. The server analyzes the text data and extracts the business flow steps.

[0479] The NLP module analyzes the text and extracts business flow steps.

[0480] 8. The server generates an initial business flow diagram from the extracted steps.

[0481] Create a flow diagram using a BPMN tool.

[0482] 9. The server sends the generated business flow diagram to the terminal.

[0483] Example prompt

[0484] "Please explain the monthly closing process for the accounting department."

[0485] "How do I add a missing step to my workflow diagram?"

[0486] "Based on this workflow, what challenges do you foresee?"

[0487] "Make a video based on the generated slides."

[0488] The above is an embodiment of the present invention. This allows even users without specialized knowledge to easily and quickly create and update workflows, thereby improving actual work efficiency. It also enables adaptive proposals that take into account the user's emotional data and the efficient creation of presentation materials and videos.

[0489] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0490] Step 1:

[0491] The user verbally explains the workflow.

[0492] Input: The user's verbal description. For example, "I'm going to explain the monthly closing process for the accounting department. First, I'll collect all the transaction data..."

[0493] Output: User's voice data.

[0494] Step 2:

[0495] The device records the user's voice.

[0496] Specific operation: The device's microphone captures the sound, and the recording module is activated to generate digital audio data.

[0497] Input: User's voice.

[0498] Output: Recorded audio data (WAV or MP3 format).

[0499] Step 3:

[0500] The device sends the recorded voice data to the voice recognition engine and emotion engine.

[0501] What happens: The recorded voice data is converted into an appropriate format and sent as a request to a speech recognition engine (e.g., Google Cloud Speech-to-Text API) and an emotion engine (e.g., IBM Watson Tone Analyzer API).

[0502] Input: Recorded audio data.

[0503] Output: The request data sent to the speech recognition engine and emotion engine.

[0504] Step 4:

[0505] A voice recognition engine converts the voice data into text data.

[0506] What it does: The speech recognition engine analyzes the speech waveform and generates corresponding text.

[0507] Input: Audio data.

[0508] Output: Text data (user's explanation).

[0509] Step 5:

[0510] The emotion engine recognizes the user's emotion from the voice data.

[0511] Specific operation: The emotion analysis module analyzes the voice features and outputs the user's emotional state (e.g., "excited" or "calm").

[0512] Input: Audio data.

[0513] Output: Emotion data (emotional state).

[0514] Step 6:

[0515] A server receives the text data and emotion data obtained from the speech recognition engine and emotion engine.

[0516] Specific operation: Receives the API response and stores the text data and emotion data in the analytical storage (database).

[0517] Input: Text data, emotion data.

[0518] Output: Data stored in analytical storage.

[0519] Step 7:

[0520] The server analyzes the text data and extracts the steps of the business flow.

[0521] Specific operation: The NLP module analyzes the text data and extracts each step of the business flow (e.g., "collect transaction data" and "verify data").

[0522] Input: Text data.

[0523] Output: A step in the business flow.

[0524] Step 8:

[0525] The server generates an initial business flow diagram from the extracted steps.

[0526] Specific operation: Create a business flow diagram using a BPMN tool and arrange the extracted steps as nodes and edges.

[0527] Input: A workflow step.

[0528] Output: Business flow diagram.

[0529] Step 9:

[0530] The server sends the generated business flow diagram to the terminal.

[0531] Specific operation: The business flow diagram is converted into an image (e.g. PNG) or XML format and sent to the terminal as an HTTP response.

[0532] Input: Business flow diagram.

[0533] Output: The workflow diagram sent to the terminal.

[0534] Step 10:

[0535] The user checks the workflow diagram and verbally provides additional information or corrections.

[0536] Input: The displayed business flow diagram, additional information or corrections made verbally by the user.

[0537] Output: Audio data including any user additions or corrections.

[0538] Step 11:

[0539] The device records the audio again and sends it to the speech recognition engine and emotion engine.

[0540] Specific operation: The recording module starts again, captures the user's voice, and generates digital voice data, which is then sent to the speech recognition engine and emotion engine.

[0541] Input: User spoken additions and corrections.

[0542] Output: Speech data sent to the speech recognition engine and emotion engine.

[0543] Step 12:

[0544] The voice recognition engine converts the voice data into text data again, which is received by the server.

[0545] Specific operation: The speech recognition engine analyzes the speech waveform and generates corresponding text. The generated text data is sent to the server, which receives it and stores it in a database.

[0546] Input: Audio data.

[0547] Output: Text data (additional information and corrections).

[0548] Step 13:

[0549] The emotion engine again recognizes the user's emotion from the voice data, and the data is received by the server.

[0550] Specific operation: The emotion analysis module analyzes the voice features and outputs a new emotional state. The generated emotional data is sent to the server and stored in a database.

[0551] Input: Audio data.

[0552] Output: Emotional data (emotional state in response to additional information or corrections).

[0553] Step 14:

[0554] The server updates the workflow diagram based on the information and emotion data acquired again.

[0555] Specific operation: Incorporate new steps (additional information or modifications) into an existing flow diagram, and change the diagram's color scheme and node display format based on emotion data.

[0556] Input: Text data of additional information and corrections, emotion data, and existing business flow diagram.

[0557] Output: An updated workflow diagram.

[0558] Step 15:

[0559] The server sends the updated business process diagram to the terminal.

[0560] Specific operation: The regenerated diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[0561] Input: Updated business flow diagram.

[0562] Output: The updated workflow diagram sent to the terminal.

[0563] Step 16:

[0564] The server automatically analyzes common problems and their solutions based on the generated business flow diagram.

[0565] Specific operation: The problem extraction module analyzes the business flow diagram and identifies potential problems (e.g., "insufficient data verification") from data patterns. The solution proposal module takes into account emotional data and proposes optimal solutions (e.g., "standardize data verification procedures").

[0566] Input: Updated workflow diagram, emotion data.

[0567] Output: Challenges and their solutions.

[0568] Step 17:

[0569] The terminal allows the user to confirm the presented problem and solution.

[0570] What it does: Display the problem and solution on the screen and allow the user to review it in the way that is most acceptable to them (e.g., read it out loud in a quiet voice).

[0571] Input: Challenges and solutions.

[0572] Output: The problem and solution presented to the user.

[0573] Step 18:

[0574] The user checks the issues and solutions and instructs corrections as necessary.

[0575] Input: The presented problem and solution.

[0576] Output: User feedback.

[0577] Step 19:

[0578] The server automatically generates presentation slides based on the business flow diagram, issues, and solutions.

[0579] How it works: The slide generation module uses Google Slides APIs and other tools to embed business flow diagrams, challenges, and solutions into templates, and applies emotionally-sensitive layouts and designs.

[0580] Input: Business flow diagram, problem, solution, sentiment data.

[0581] Output: Slides for presentation.

[0582] Step 20:

[0583] The server creates a video based on the generated slides and sends it to the terminal.

[0584] Specific operation: The video generation module uses APIs such as Adobe Premiere Pro to animate slides and add narration. It adjusts the tone and speed of the narration based on emotional data. The generated video file is sent to the device as an HTTP response.

[0585] Input: Presentation slides, emotion data.

[0586] Output: Video file.

[0587] Step 21:

[0588] Users can review the final slides and videos and download or share them as desired.

[0589] Specific behavior: The device displays the slides and video, and the user reviews them. If necessary, they are saved to local storage or shared with other users.

[0590] Input: Slides and videos.

[0591] Output: Verified slides and videos.

[0592] (Application example 2)

[0593] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0594] Conventional workflow creation systems require a great deal of effort, as they convert users' verbal descriptions into text data, then manually analyze that text data and generate a workflow diagram. Furthermore, if additional information or corrections are needed in the generated workflow diagram, they must be done manually again, which is inefficient. Furthermore, there was no function to automatically analyze and present issues and solutions based on the generated workflow diagram, so separate analysis work was required. Furthermore, conventional systems had difficulty recognizing users' emotions and providing adaptive information and suggestions, which prevented them from improving the user experience.

[0595] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0596] In this invention, the server includes: a means for converting a workflow described orally by a user into text data using speech recognition technology; a means for analyzing the text data to extract workflow steps and generate a workflow diagram; a means for converting verbal corrections and supplementary information added by the user into text data using speech recognition technology again to update the workflow diagram; a means for automatically analyzing and presenting problems and solutions based on the generated workflow diagram; a means for automatically generating a presentation document based on the workflow diagram, problems, and solutions; a means for creating visual materials based on the automatically generated document; a means for recognizing emotions from the user's voice; and a means for adaptively presenting information and making suggestions based on the emotion data. This enables efficient conversion of a workflow described orally by a user into text data and automatic generation and updating of a workflow diagram. Furthermore, the automatic presentation of problems and solutions improves the efficiency of the analysis process, and adaptive information presentation and suggestions based on the user's emotions significantly improves the user experience.

[0597] "Voice recognition technology" is a technology that converts voice into digital data and then into text data.

[0598] "Text data" refers to text information converted using voice recognition technology.

[0599] A "business flow" is a description of the steps or procedures of a particular business process.

[0600] A "workflow diagram" is a diagram that visually represents a workflow.

[0601] "Emotion recognition" is a technology that analyzes and recognizes a user's emotional state from their voice and other data.

[0602] A "means" is a specific method or device for achieving a specific purpose.

[0603] "User" refers to a person who uses the system.

[0604] An "issue" is a potential problem or obstacle in the workflow.

[0605] A "solution" is a method or means for addressing a problem.

[0606] A "presentation document" is a document that systematically summarizes business processes, issues, and solutions.

[0607] "Visual materials" are materials that are easy to understand visually and are created based on presentation documents.

[0608] "Adaptive information presentation" means providing information in an optimal form based on the user's emotional data.

[0609] MODE FOR CARRYING OUT THE INVENTION

[0610] This invention provides a system that converts a user's verbal description of a workflow into text data using speech recognition technology, analyzes the text data to extract workflow steps, and generates a workflow diagram. Furthermore, this system incorporates emotion recognition functionality, allowing it to make adaptive suggestions and present information based on the user's emotional state.

[0611] Speech and Emotion Recognition

[0612] 1. The user verbally explains the workflow. For example, "I'll explain the monthly closing process for the accounting department. First, we collect all transaction data..."

[0613] 2. The device records the user's voice using a microphone.

[0614] 3. The device sends the recorded voice data to a speech recognition engine and an emotion engine. The speech recognition engine uses Google's speech recognition API, and the emotion engine uses the Transformers sentiment-analysis pipeline.

[0615] 4. The speech recognition engine converts the speech data into text data, which is then generated.

[0616] 5. The emotion engine recognizes the user's emotions from the voice data and generates emotion data.

[0617] Business flow diagram generation and emotional adaptation

[0618] 6. The server analyzes the acquired text data and extracts the business flow steps using a Natural Language Processing (NLP) module.

[0619] 7. The server generates an initial business flow diagram from the extracted steps. The flow diagram is created using the business process model.

[0620] 8. The server sends the generated business flow diagram to the terminal. The diagram is sent as an image or in XML format.

[0621] Completion of missing information and emotional adaptation

[0622] 9. The user checks the workflow diagram and verbally provides additional information or corrections if necessary. For example, "There's a data check step missing here."

[0623] 10. The device records the audio again and sends it to the speech recognition engine and emotion engine.

[0624] 11. The voice recognition engine converts the voice data back into text data, which is received by the server.

[0625] 12. The server updates the workflow diagram based on the information and emotion data it has acquired again.

[0626] Presenting the problem and solution

[0627] 13. The server automatically analyzes and presents common problems and their solutions based on the generated business flow diagram.

[0628] 14. The device prompts the user to confirm the presented problem and solution.

[0629] Slide and video generation

[0630] 15. The server automatically generates presentation documents based on the business flow diagram, issues, and solutions. It uses a slide generation module.

[0631] 16. The server creates visual materials based on the generated documents and sends them to the device. The visual materials can be in the form of PowerPoint slides or videos.

[0632] Examples of concrete examples and prompts

[0633] As an example, a user might verbally describe:

[0634] "I will explain how to start and stop the production line. First, start the machine and then perform a line check. This is where the machine inspection step is required," he says.

[0635] Based on this prompt, the system will instruct the user as follows:

[0636] Please describe your workflow in the following format:

[0637] 1. First step explanation

[0638] 2. Next Steps Explained

[0639] 3. Next step explanation...

[0640] For example, 'First, start the machine, then do a line check...'"

[0641] As described above, the present invention provides a system that allows a user to explain a workflow and incorporates it efficiently into a workflow diagram using speech recognition and emotion recognition technology, thereby supporting problem solving and the creation of presentation materials.

[0642] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0643] Step 1:

[0644] The user verbally explains the workflow. The input is the workflow information in the form of the user's voice. For example, the user might say, "I'll explain the monthly closing process for the accounting department. First, I'll collect all the transaction data..." The output is the user's voice.

[0645] Step 2:

[0646] The device records the user's voice. The input is the user's voice. The device's microphone is used to capture the voice and store it as digital audio data. The output is the recorded audio data.

[0647] Step 3:

[0648] The device sends the recorded voice data to the speech recognition engine and emotion engine. The input is the recorded voice data. The voice data is converted into an appropriate format and a request is sent to the engine API. The output is an API request to the speech recognition engine and emotion engine.

[0649] Step 4:

[0650] A speech recognition engine converts speech data into text data. The input is speech data. The engine analyzes the speech waveform and generates text output. The output is text data.

[0651] Step 5:

[0652] The emotion engine recognizes the user's emotion from the voice data. The input is the voice data. The emotion analysis module analyzes the voice features and outputs the emotional state. The output is emotion data.

[0653] Step 6:

[0654] The server receives the text data and emotion data obtained from the speech recognition engine and emotion engine. The input is the text data and emotion data. The server receives the API response and stores the text data and emotion data in the storage for analysis. The output is the stored text data and emotion data.

[0655] Step 7:

[0656] The server analyzes text data and extracts workflow steps. The input is stored text data. The Natural Language Processing (NLP) module analyzes the text and extracts workflow steps. The output is the extracted workflow steps.

[0657] Step 8:

[0658] The server generates an initial business flow diagram from the extracted steps. The input is the extracted business flow steps. The flow diagram is created using the business process model. The output is the initial business flow diagram.

[0659] Step 9:

[0660] The server sends the generated business flow diagram to the terminal. The input is the initial business flow diagram. The flow diagram is converted into image or XML format and sent to the terminal as an HTTP response. The output is the business flow diagram sent to the terminal.

[0661] Step 10:

[0662] The user checks the workflow diagram and verbally provides additional information or corrections if necessary. The input is the workflow diagram and the user's voice. The user verbally provides supplementary information, for example, "There's a data check step missing here." The output is the user's voice, including the supplementary information.

[0663] Step 11:

[0664] The device records the audio again and sends it to the speech recognition engine and emotion engine. The input is the audio data including supplementary information. The recording module captures the audio and sends it to the speech recognition engine and emotion engine. The output is an API request to the speech recognition engine and emotion engine.

[0665] Step 12:

[0666] The speech recognition engine converts the speech data back into text data, which is received by the server. The input is speech data that includes supplemental information. The engine analyzes the speech waveform and generates text output, and the server receives the API response and stores it in the analysis storage. The output is text data that includes the stored supplemental information.

[0667] Step 13:

[0668] The emotion engine again recognizes the user's emotion from the voice data, which is received by the server. The input is voice data including supplementary information. The emotion analysis module analyzes the voice features and outputs the emotional state. The server receives the API response and stores it in the analysis storage. The output is the stored added emotion data.

[0669] Step 14:

[0670] The server updates the workflow diagram based on the newly acquired information and emotion data. The input is the new step and emotion data. The new step is incorporated into the existing workflow diagram, and the diagram is adaptively regenerated based on the emotion data. The output is the updated workflow diagram.

[0671] Step 15:

[0672] The server automatically generates a presentation document based on the generated business flow diagram, issues, and solutions. The input is the business flow diagram, issues, and solutions. The slide generation module embeds the business flow diagram, issues, and solutions in a template and applies an emotionally sensitive layout and design. The output is a presentation document.

[0673] Step 16:

[0674] The server creates visual materials based on the generated documents and sends them to the terminal. The input is the presentation document. The server animates the visual materials, adds narration, generates a video file, and sends it to the terminal as an HTTP response. The output is the visual materials.

[0675] The above is a detailed description of the specific process steps for carrying out the invention.

[0676] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0677] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0678] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0679] [Second embodiment]

[0680] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0681] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0682] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0683] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0684] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0685] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0686] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0687] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0688] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0689] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0690] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0691] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0692] This invention relates to a system that converts a user's verbal description of a workflow into text data using speech recognition technology, analyzes the text data to extract workflow steps, and generates a workflow diagram. This system also has the ability to convert verbal corrections and supplementary information added by the user back into text data using speech recognition technology, updating the workflow diagram. Furthermore, based on the generated workflow diagram, the system automatically analyzes and presents issues and their solutions. It also automatically generates presentation slides based on the workflow diagram, issues, and solutions, and creates videos based on those slides.

[0693] Explaining program processing in natural language

[0694] Voice Recognition

[0695] 1. The user verbally explains the workflow. For example, "I'd like to explain the monthly closing process for the accounting department. First, we collect all transaction data..."

[0696] 2. The device records the user's voice.

[0697] The recording module will be activated and the microphone will capture your voice.

[0698] 3. The device sends the recorded voice data to the voice recognition engine.

[0699] Converts the audio data into the appropriate format and sends the request to the Engine API.

[0700] 4. The voice recognition engine converts the voice data into text data.

[0701] The engine analyzes the audio waveform and generates text output.

[0702] 5. The server receives the text data obtained from the speech recognition engine.

[0703] The API response is received and the text data is stored in the analysis storage.

[0704] Generate a workflow diagram

[0705] 6. The server analyzes the text data obtained from the speech recognition engine.

[0706] The Natural Language Processing (NLP) module analyzes the text and extracts business flow steps.

[0707] 7. The server generates an initial business flow diagram from the extracted steps.

[0708] Create flow diagrams using BPMN (Business Process Model and Notation) tools.

[0709] 8. The server sends the generated business flow diagram to the terminal.

[0710] The flow diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[0711] Completion of missing information

[0712] 9. The user checks the workflow diagram and verbally provides additional information or corrections if necessary. For example, "There's a data check step missing here."

[0713] 10. The device records the audio again and sends it to the speech recognition engine.

[0714] The recording module will be activated again and the microphone will capture the audio.

[0715] The recording data is sent to the speech recognition engine.

[0716] 11. The speech recognition engine converts the speech back into text data, which is then received by the server.

[0717] As before, we analyze the audio waveform and generate text output.

[0718] Receive the API response and store it as additional information in the analytical storage.

[0719] 12. The server updates the business flow diagram based on the information obtained again.

[0720] Incorporate the new steps into an existing flow diagram and regenerate the updated diagram.

[0721] 13. The server sends the updated workflow diagram to the terminal.

[0722] The regenerated diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[0723] Presenting the problem and solution

[0724] 14. The server automatically analyzes common problems and their solutions based on the generated business flow diagram.

[0725] The issue extraction module analyzes the flow diagram and identifies potential issues from data patterns.

[0726] The solution proposal module searches for suitable solutions from a historical database and generates proposals.

[0727] 15. The terminal prompts the user to confirm the presented problem and solution.

[0728] The problem and solution are displayed on the screen so that the user can check it.

[0729] 16. The user reviews the problem and solution and suggests modifications if necessary. For example, "This solution doesn't apply here, so let's think of another way."

[0730] Slide and video generation

[0731] 17. The server automatically generates presentation slides based on the business flow diagram, issues, and solutions.

[0732] The slide generation module embeds business flow diagrams, challenges, and solutions into templates.

[0733] 18. The server creates a video based on the generated slides and sends it to the terminal.

[0734] A video generation module animates the slides and provides narration.

[0735] A video file is generated and sent to the device as an HTTP response.

[0736] 19. The user reviews the final slides and video and downloads or shares them as desired.

[0737] The device displays the slides and video, which the user can review.

[0738] Specific examples

[0739] 1. The user says, "I'd like to explain the monthly closing process for the accounting department. First, we'll collect all the transaction data..."

[0740] 2. The device records the audio and converts it into text using a speech recognition engine.

[0741] 3. The server extracts steps such as "collect transaction data" and generates an initial business flow diagram.

[0742] 4. The server sends the generated diagram to the terminal, where the user can confirm it.

[0743] 5. The user verbally provides additional information, such as, "There's a data check step missing here."

[0744] 6. The device will record the audio again and convert it into text using a speech recognition engine.

[0745] 7. The server generates an updated workflow diagram with the new information added.

[0746] 8. Based on the diagram, the server provides information such as "common issues in data checking" and "solutions," and automatically generates slides.

[0747] 9. Finally, the presentation slides and video are sent to the device for the user to review.

[0748] The above is a detailed description of the embodiment of the present invention, which enables effective BPR without specialized knowledge and promotes in-house development of business processes.

[0749] The processing flow will be explained below.

[0750] Step 1:

[0751] The user verbally explains the workflow. For example, "I will explain the monthly closing process for the accounting department. First, we collect all transaction data..."

[0752] Step 2:

[0753] The terminal records the user's voice.

[0754] The recording module will be activated and the microphone will capture your voice.

[0755] Step 3:

[0756] The device sends the recorded voice data to a voice recognition engine.

[0757] Converts the audio data into the appropriate format and sends the request to the Engine API.

[0758] Step 4:

[0759] A voice recognition engine converts the voice data into text data.

[0760] The engine analyzes the audio waveform and generates text output.

[0761] Step 5:

[0762] The server receives the text data obtained from the speech recognition engine.

[0763] The API response is received and the text data is stored in the analysis storage.

[0764] Step 6:

[0765] The server analyzes the text data and extracts the steps of the business flow.

[0766] The Natural Language Processing (NLP) module analyzes the text and extracts business flow steps.

[0767] Step 7:

[0768] The server generates an initial business flow diagram from the extracted steps.

[0769] Create flow diagrams using BPMN (Business Process Model and Notation) tools.

[0770] Step 8:

[0771] The server sends the generated business flow diagram to the terminal.

[0772] The flow diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[0773] Step 9:

[0774] The user checks the workflow diagram and verbally provides additional information or corrections if necessary. For example, "There's a data check step missing here."

[0775] Step 10:

[0776] The device will record the audio again and send it to the speech recognition engine.

[0777] The recording module will be activated again and the microphone will capture the audio.

[0778] The recording data is sent to the speech recognition engine.

[0779] Step 11:

[0780] The voice recognition engine converts the voice data into text data again, which is received by the server.

[0781] As before, we analyze the audio waveform and generate text output.

[0782] Receive the API response and store it as additional information in the analytical storage.

[0783] Step 12:

[0784] The server updates the business flow diagram based on the information retrieved again.

[0785] Incorporate the new steps into an existing flow diagram and regenerate the updated diagram.

[0786] Step 13:

[0787] The server sends the updated business process diagram to the terminal.

[0788] The regenerated diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[0789] Step 14:

[0790] The server automatically analyzes common problems and their solutions based on the generated business flow diagram.

[0791] The issue extraction module analyzes the flow diagram and identifies potential issues from data patterns.

[0792] The solution proposal module searches for suitable solutions from a historical database and generates proposals.

[0793] Step 15:

[0794] The terminal allows the user to confirm the presented problem and solution.

[0795] The problem and solution are displayed on the screen so that the user can check it.

[0796] Step 16:

[0797] The user reviews the problem and solution and suggests modifications if necessary, for example, "This solution doesn't apply here, so let's think of another way."

[0798] Step 17:

[0799] The server automatically generates presentation slides based on the business flow diagram, issues, and solutions.

[0800] The slide generation module embeds business flow diagrams, challenges, and solutions into templates.

[0801] Step 18:

[0802] The server creates a video based on the generated slides and sends it to the terminal.

[0803] A video generation module animates the slides and provides narration.

[0804] A video file is generated and sent to the device as an HTTP response.

[0805] Step 19:

[0806] Users can review the final slides and videos and download or share them as desired.

[0807] The device presents slides and videos, which the user reviews.

[0808] The above is a detailed description of the specific processing steps of the program.

[0809] Example 1

[0810] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0811] Traditionally, the process of formulating workflows was largely manual, requiring time and effort. Furthermore, reviewing and improving workflows required extensive specialized knowledge and skills, making the process inefficient. Furthermore, updating workflow diagrams, identifying issues, and proposing solutions were complex, creating a need for automation.

[0812] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0813] In this invention, the server includes means for converting a business process described orally by a user into text data using speech recognition technology, means for analyzing the text data to extract business process steps and generate a business process diagram, means for converting verbal corrections and supplementary information added by the user into text data again using speech recognition technology and updating the business process diagram, means for automatically analyzing and presenting issues and their solutions based on the generated business process diagram, means for automatically generating presentation slides based on the business process diagram, issues, and solutions, and means for creating videos based on the automatically generated slides. This enables effective improvement of business processes without specialized knowledge and promotes in-house development of business processes.

[0814] A "user" is someone who uses the system to explain the workflow and add corrections or supplementary information.

[0815] "Speech recognition technology" is a technology that analyzes the voice data of a user's oral explanation and converts it into text data.

[0816] "Text data" refers to the text information of voice data converted using voice recognition technology.

[0817] "Analysis" refers to the process of classifying text data, interpreting its meaning and structure, and extracting business flow steps.

[0818] A "business flow step" refers to an individual task or processing stage in a business process.

[0819] A "business flow diagram" is a diagram that visually represents the steps of a business flow.

[0820] An "issue" is a problem or obstacle that occurs in a business flow.

[0821] A "solution" is a method or means for solving a particular problem.

[0822] "Presentation slides" are materials used to visually present information such as business flow diagrams, issues, and solutions.

[0823] "Video" refers to dynamic video content created by adding animation and narration to automatically generated slides.

[0824] This invention relates to a system that uses speech recognition technology to generate a workflow diagram based on a user's verbal description of the workflow, thereby improving the efficiency of business processes. Specifically, the system converts the speech data into text data, analyzes it, extracts the workflow steps, and automatically generates a workflow diagram. Furthermore, the system is characterized by the fact that the workflow diagram is constantly updated with the latest information as the user verbally provides additional information or corrections, and automatically presents issues and their solutions.

[0825] To implement this system, the following hardware and software are required:

[0826] Device: A device used to record audio and communicate with the server. Examples include smartphones, tablets, and computers.

[0827] Server: The central computer system that provides the speech recognition and text analysis technologies. This may include a cloud server or a dedicated server.

[0828] Speech recognition engine: Software that converts a user's voice into text data. Representative engines include Google Cloud Speech-to-Text API and IBM Watson Speech to Text API.

[0829] Natural Language Processing (NLP) module: A software module for analyzing text data and extracting business flow steps.

[0830] BPMN (Business Process Model and Notation) tool: A dedicated tool for generating business flow diagrams.

[0831] Data analysis module: Software for automatically analyzing issues and their solutions from business flow diagrams.

[0832] Slide generation module: A module for automatically generating slides for presentations.

[0833] Video Generation Module: A module for creating videos based on generated slides.

[0834] The specific operating procedure of the system is as follows: first, the user verbally explains the workflow, which is then recorded by the device. The recorded voice data is sent to the server, where it is converted into text data by a speech recognition engine. Next, the server analyzes the text data using an NLP module and extracts the workflow steps. A BPMN tool generates a workflow diagram based on the extracted steps.

[0835] If the user verbally adds or modifies information to the diagram, the device records the voice again and sends it to a speech recognition engine to convert it into text data. The server updates the workflow diagram based on the new information and automatically analyzes and presents the issues and solutions. The server then uses a slide generation module to create presentation slides based on the workflow diagram, issues, and solutions. Furthermore, the server uses a video generation module to create a video based on the created slides and sends it to the device.

[0836] Specific examples

[0837] The user verbally explains, "I'd like to explain the monthly closing process for the accounting department. First, we'll collect all transaction data..." The device records the speech and converts it into text data using a voice recognition engine. The server extracts steps such as "collect transaction data" and generates an initial business flow diagram. The server sends the diagram it has generated to the device, where the user can confirm it.

[0838] The user verbally provides additional information, saying, "There's a data check step missing here." The device records the voice again and converts it into text data using a speech recognition engine. The server adds the new information and generates an updated business flow diagram. Based on the diagram, the server provides "common issues in data checks" and "solutions," and automatically generates slides. Finally, the presentation slides and video are sent to the device, where the user can review them.

[0839] The above is a detailed description of the embodiment of the present invention. This system enables effective improvement of business processes without specialized knowledge, and promotes automation and in-house production of business flows.

[0840] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0841] Step 1:

[0842] The user verbally explains the workflow. For example, "I'll explain the monthly closing process for the accounting department. First, we'll collect all transaction data..." This explanation becomes the initial data input.

[0843] Step 2:

[0844] The terminal records the user's voice. The recording module is activated and the device's microphone captures the voice. This captured voice data becomes the input data for the next step.

[0845] Step 3:

[0846] The device sends the recorded voice data to the speech recognition engine. The voice data is converted to an appropriate format (e.g., WAV, FLAC) and sent to the speech recognition engine using an HTTP request. This makes the voice data the target data for conversion.

[0847] Step 4:

[0848] The speech recognition engine converts the speech data into text data. It analyzes the speech waveform and generates text output. For example, the text data generated is "Collecting transaction data." This text data becomes the input data for analysis.

[0849] Step 5:

[0850] The server receives the text data obtained from the speech recognition engine. The JSON-formatted text data is received as an API response and stored in the analysis storage. This stored data becomes the input data for subsequent analysis processing.

[0851] Step 6:

[0852] The server analyzes the text data obtained from the speech recognition engine. The NLP module analyzes the text data, extracts nouns and verbs, and identifies the steps in the business flow. For example, steps such as "collect transaction data" and "verify data" are extracted. The results of this analysis become the input data for generating the flow diagram.

[0853] Step 7:

[0854] The server extracts the business flow steps from the analysis results. The NLP module lists the steps obtained and clarifies the order and relationships required to generate a business flow diagram. This becomes the preparatory data for generating the flow diagram.

[0855] Step 8:

[0856] The server generates a business flow diagram based on the extracted steps. Using a BPMN tool, a business flow diagram is created that visually represents a series of steps. For example, a flow diagram of "Collect transaction data" → "Confirm data" → "Complete monthly closing process" is generated. This generated flow diagram becomes the input data for the next import.

[0857] Step 9:

[0858] The server sends the generated business flow diagram to the terminal. The flow diagram is converted into an image or XML format and sent to the terminal as an HTTP response. This allows the user to check the specific flow diagram.

[0859] Step 10:

[0860] The user checks the workflow diagram and verbally provides additional information or corrections if necessary. For example, they might say, "There's a data check step missing here." This additional information becomes the input data for the next step.

[0861] Step 11:

[0862] The device records the audio again and sends it to the speech recognition engine. The recording module is then activated again, capturing additional audio, converting it to the appropriate format, and sending it to the engine, which then uses it as input data for reconversion.

[0863] Step 12:

[0864] The speech recognition engine converts the speech into text data again. It analyzes the additional speech and generates text containing "data check." This text data becomes the input data for analysis.

[0865] Step 13:

[0866] The server receives and analyzes the new text data. It analyzes the updated text data and recognizes the new step "Data Check." This updated data becomes the updated input data for the flow diagram.

[0867] Step 14:

[0868] The server generates an updated business flow diagram. The new steps are incorporated into the existing flow diagram, and the flow diagram is regenerated in the format of "Collect transaction data" → "Data check" → "Data confirmation" → "Monthly closing process completed." This flow diagram will be the updated data.

[0869] Step 15:

[0870] The server sends the updated business flow diagram to the terminal. The regenerated flow diagram is converted back to PNG or SVG format and sent to the terminal. The user can then review it.

[0871] Step 16:

[0872] The server analyzes the issues and solutions based on the workflow diagram. The data analysis module analyzes the workflow diagram and identifies issues at specific steps. For example, an issue such as "data checking takes a long time" is extracted. This issue and solution become the input data for the next generated slide.

[0873] Step 17:

[0874] The terminal presents the problem and the solution to the user. The problem and the solution received from the server are displayed on the screen so that the user can confirm them. This allows the user to decide on the next action.

[0875] Step 18:

[0876] The user checks the problem and solution and gives instructions for corrections as necessary. For example, the user may give verbal instructions such as, "It would be better to use the current tool, so please think of another solution." This additional instruction becomes prompt data for the next step.

[0877] Step 19:

[0878] The server generates presentation slides based on the business flow diagram, issues, and solutions. The slide generation module embeds the business flow diagram, issues, and solutions into a template and generates a slide-format file. The output is a PowerPoint or PDF.

[0879] Step 20:

[0880] The server generates a video based on the slides and sends it to the device. The video generation module animates the generated slides, synthesizes the narration from the text, and creates a video file. This video becomes the final output data for the user.

[0881] Step 21:

[0882] The user reviews the final slides and video, and downloads or shares them as needed. The device displays the slides and video, allowing the user to review them and proceed to the next step in the process.

[0883] (Application example 1)

[0884] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0885] Optimizing business processes in factories is important from the perspective of improving efficiency and productivity on-site, but conventional methods require specialized knowledge and require a great deal of time and effort. In particular, it has been difficult to grasp the real-time progress of work within a factory and immediately present optimization proposals. In addition, organizing business flows and creating presentation materials also require time and effort, which leads to a decline in work efficiency. For this reason, there is a need for a system that can efficiently manage business processes within a factory and make immediate proposals.

[0886] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0887] In this invention, the server includes: means for converting a work flow verbally explained by a user into text data using voice recognition technology; means for analyzing the text data to extract workflow steps and generate a workflow diagram; means for converting verbal corrections and supplementary information added by the user into text data using voice recognition technology again to update the workflow diagram; means for automatically analyzing and presenting problems and solutions based on the generated workflow diagram; means for automatically generating presentation images based on the workflow diagram, problems, and solutions; means for creating videos based on the automatically generated images; means for using a robot equipped with a voice recognition module that allows a user to verbally explain factory operations and work instructions in real time; and means for analyzing voice data recorded by the robot and providing optimization suggestions based on the workflow diagram. This enables efficient management of factory business processes and real-time optimization suggestions on-site.

[0888] "User" refers to a person who uses the system to verbally explain the workflow and provide corrections and supplementary information.

[0889] "Verbal explanation" refers to the act of using voice to communicate the flow of work and correction information.

[0890] "Voice recognition technology" refers to the technology that analyzes voice and converts it into text data.

[0891] "Text data" refers to text data converted from speech using speech recognition technology.

[0892] "Workflow" refers to a series of steps or procedures for accomplishing a particular task or work.

[0893] A "step" refers to an individual unit of work within a workflow.

[0894] A "workflow diagram" refers to a visual representation of a workflow.

[0895] A "problem" refers to a glitch or inefficiency that occurs within a workflow.

[0896] A "solution" refers to a specific method or proposal for solving a problem.

[0897] "Presentation images" refers to diagrams and presentation slides that visually illustrate workflows, business problems, and solutions.

[0898] "Video creation" refers to the process of creating visual explanations in video format based on presentation images.

[0899] A "robot equipped with a voice recognition module" refers to a robot that has the ability to record voice data and analyze the voice in real time.

[0900] "Optimization proposals based on workflow diagrams" refers to proposals that analyze workflow diagrams to identify problems and show how to improve them.

[0901] "Real-time" refers to processing and providing information immediately, without delay.

[0902] This invention is a system for efficiently managing work processes in factories and making optimization proposals in real time. This system uses voice recognition technology to convert a user's verbal explanation into text data, analyzes that data to generate a workflow diagram, and finally creates images and videos for presentations based on the proposals.

[0903] System Overview

[0904] The system includes the following elements:

[0905] 1. Robot equipped with a voice recognition module

[0906] It records factory operations and work instructions in real time and converts user voice into text data.

[0907] Specific hardware: robot, microphone

[0908] Software used: Google Speech-to-Text API

[0909] 2. Server

[0910] Analyzes character data and extracts workflow steps.

[0911] Generate workflow diagrams to analyze and present problems and their solutions.

[0912] Automatically generate images and videos for presentations.

[0913] Software used: nltk, pydotplus, PaddleOCR

[0914] System action

[0915] Voice Recognition

[0916] The robot embedded in the device records work instructions and explanations in real time within the factory and converts them into text data using the Google Speech-to-Text API, which the device then sends to a server.

[0917] Text data analysis

[0918] The server receives the text data and uses natural language processing tools such as nltk to extract each step of the workflow. Pydotplus is used to generate a workflow diagram from the extracted steps.

[0919] Presenting the problem and solution

[0920] Based on the generated workflow diagram, the server automatically analyzes potential problems, searches for appropriate solutions from past data, and presents them to the user. This analysis utilizes data pattern mining technology.

[0921] Image and video generation for presentations

[0922] The server generates presentation images and explanatory videos based on the generated workflow diagram and analysis results. The presentation images are generated by embedding information in templates, and the explanatory videos are generated by animating the images and adding narration.

[0923] Specific examples

[0924] 1. The user says, "I will explain the assembly procedure for the production line. First, prepare all the parts, then connect part A to part B. After that, perform a quality check..."

[0925] 2. The device records the audio and converts it into text using the Google Speech-to-Text API.

[0926] 3. The server receives the text data and extracts steps such as "preparing parts," "connecting parts A and B," and "quality check."

[0927] 4. The server generates a workflow diagram based on the extracted information and displays it on the terminal for the user to review.

[0928] 5. The user verbally adds additional information, saying, "After the quality check, there will be packaging."

[0929] 6. The device records the audio again and converts it into text data using the Google Speech-to-Text API.

[0930] 7. The server takes in the new information and generates an updated workflow diagram.

[0931] 8. Based on this workflow diagram and analysis results, the server provides optimization suggestions and generates presentation images and explanatory videos.

[0932] Prompt Sentence Examples

[0933] "Please explain in detail the workflow of your factory's production line. Then, please add any missing steps or areas for improvement. Also, please add details of each process."

[0934] This system allows users to effectively optimize factory operations without specialized knowledge, making it possible to improve the efficiency of work flows.

[0935] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0936] Step 1:

[0937] The terminal records the work instructions and workflows that users verbally explain in the factory. Specifically, a microphone installed on the robot captures the user's voice, and the recording module converts the voice into a digital format. The recording data is generated and sent to the server. The output based on the input is digital voice data.

[0938] Step 2:

[0939] The server sends the received digital voice data to the Google Speech-to-Text API, which converts the voice into text data (character data). The server then calls the API of the speech recognition engine and converts the voice input into text format. The input data is voice data, and the output data is character data.

[0940] Step 3:

[0941] The server analyzes the text data and extracts each step of the workflow. Specifically, the Natural Language Processing (NLP) module analyzes the text data and identifies a series of work steps from the content. The input is the text data, and the output is a list of the analyzed work steps.

[0942] Step 4:

[0943] The server generates a workflow diagram based on the extracted work steps. The server uses pydotplus to create a flow diagram that connects each step. The input is a list of work steps, and the output is a workflow diagram. The flow diagram is generated in image or XML format and sent to the terminal.

[0944] Step 5:

[0945] The user checks the workflow diagram and provides additional verbal explanations if there is missing information or if corrections are required. Specifically, when the user explains again, the device re-records the voice and sends it to the server. The input data is the user's voice, and the output data is voice data including supplementary information.

[0946] Step 6:

[0947] The server again converts the audio data of the additional explanation into text data using the Google Speech-to-Text API and analyzes the text data as before. The NLP module analyzes the added text information and updates the existing workflow diagram. The input is the audio data containing the supplementary information, and the output is the updated text data.

[0948] Step 7:

[0949] The server regenerates the workflow diagram based on the updated work steps. It uses pydotplus to incorporate the new steps into the existing diagram and recreates the updated diagram. The input is the updated list of work steps, and the output is the updated workflow diagram. The updated diagram is also sent to the terminal.

[0950] Step 8:

[0951] The server automatically analyzes and presents potential problems and their solutions based on the workflow diagram. Using a problem extraction module and a solution presentation module, the server analyzes the generated workflow diagram, identifies problems, and searches a database for appropriate solutions. The input is the workflow diagram, and the output is a list of problems and solutions, which is also sent to the terminal.

[0952] Step 9:

[0953] The server automatically generates images and videos for presentations based on the workflow diagram, problems, and solutions. The server uses a slide generation module and a video generation module to embed information in templates and create slides and videos. The input is the workflow diagram, problems, and a list of solutions, and the output is the images and video files for presentations. Finally, these are also sent to the terminal for the user to review.

[0954] Each step is explained in detail to help you understand how the system streamlines factory operations in real time.

[0955] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0956] This invention relates to a system that converts a user's verbal description of a workflow into text data using speech recognition technology, analyzes the text data to extract workflow steps, and generates a workflow diagram. This system also has the ability to convert verbal corrections and supplementary information added by the user back into text data using speech recognition technology, updating the workflow diagram. Based on the generated workflow diagram, the system automatically analyzes and presents issues and their solutions. Furthermore, the system automatically generates presentation slides based on the workflow diagram, issues, and solutions, and creates videos based on those slides.

[0957] Furthermore, by incorporating an emotion engine that recognizes emotions from the user's voice, the present invention realizes adaptive suggestions and information presentation based on the user's emotional state. This emotion engine also simultaneously acquires and stores user emotion data, which can be used for later analysis and improvement.

[0958] Explaining program processing in natural language

[0959] Speech and Emotion Recognition

[0960] 1. The user verbally explains the workflow. For example, "I'll explain the monthly closing process for the accounting department. First, we collect all transaction data..."

[0961] 2. The device records the user's voice.

[0962] The recording module will be activated and the microphone will capture your voice.

[0963] 3. The device sends the recorded voice data to the speech recognition engine and emotion engine.

[0964] Converts the audio data into the appropriate format and sends the request to the Engine API.

[0965] 4. The speech recognition engine converts the voice data into text data.

[0966] The engine analyzes the audio waveform and generates text output.

[0967] 5. The emotion engine recognizes the user's emotions from the voice data.

[0968] The emotion analysis module analyzes the speech features and outputs the emotional state.

[0969] 6. The server receives the text data and emotion data obtained from the speech recognition engine and emotion engine.

[0970] The API response is received and the text data and emotion data are stored in the storage for analysis.

[0971] Business flow diagram generation and emotional adaptation

[0972] 6. The server analyzes the text data and extracts the steps of the business flow.

[0973] The Natural Language Processing (NLP) module analyzes the text and extracts business flow steps.

[0974] 7. The server generates an initial business flow diagram from the extracted steps.

[0975] Create flow diagrams using BPMN (Business Process Model and Notation) tools.

[0976] 8. The server sends the generated business flow diagram to the terminal.

[0977] The flow diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[0978] Completion of missing information and emotional adaptation

[0979] 9. The user checks the workflow diagram and verbally provides additional information or corrections if necessary. For example, "There's a data check step missing here."

[0980] 10. The device records the audio again and sends it to the speech recognition engine and emotion engine.

[0981] The recording module will be activated again and the microphone will capture the audio.

[0982] The recording data is sent to the speech recognition engine and emotion engine.

[0983] 11. The voice recognition engine converts the voice data back into text data, which is received by the server.

[0984] As before, we analyze the audio waveform and generate text output.

[0985] Receive the API response and store it as additional information in the analytical storage.

[0986] 12. The emotion engine again recognizes the user's emotion from the voice data, and the server receives it.

[0987] The emotion analysis module analyzes the speech features and outputs the emotional state.

[0988] The API response is received and the added emotion data is also stored in the storage for analysis.

[0989] 13. The server updates the workflow diagram based on the information and emotion data obtained again.

[0990] New steps are incorporated into existing flow diagrams, and the diagrams are adaptively regenerated based on emotion data.

[0991] 14. The server sends the updated business flow diagram to the terminal.

[0992] The regenerated diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[0993] Presenting the problem and solution

[0994] 15. The server automatically analyzes common problems and their solutions based on the generated business flow diagram.

[0995] The issue extraction module analyzes the flow diagram and identifies potential issues from data patterns.

[0996] The solution suggestion module also takes into account the emotional data and suggests the best solution.

[0997] 16. The device prompts the user to confirm the presented problem and solution.

[0998] The problem and solution are presented on the screen, allowing the user to see it in the way that is most receptive to their emotions.

[0999] 17. The user reviews the problem and solution and suggests modifications if necessary. For example, "This solution doesn't work here, so let's think of another way."

[1000] Slide and video generation

[1001] 18. The server automatically generates presentation slides based on the business flow diagram, issues, and solutions.

[1002] The slide generation module embeds workflow diagrams, challenges, and solutions into templates, applying emotionally sensitive layouts and designs.

[1003] 19. The server creates a video based on the generated slides and sends it to the terminal.

[1004] A video generation module animates the slides and provides narration.

[1005] Adjust the tone and speed of narration based on emotional data.

[1006] A video file is generated and sent to the device as an HTTP response.

[1007] 20. User reviews the final slides and video and downloads or shares as desired.

[1008] The device displays slides and videos, which the user reviews.

[1009] Specific examples

[1010] 1. The user says, "I'd like to explain the monthly closing process for the accounting department. First, we'll collect all the transaction data..."

[1011] 2. The device records the audio and analyzes it using a voice recognition engine and emotion engine.

[1012] 3. The server extracts steps such as "collect transaction data" and generates an initial workflow diagram. If the user's emotion is recognized as "excited," the information is presented in a calm tone.

[1013] 4. The server sends the generated diagram to the terminal, where the user can confirm it.

[1014] 5. The user verbally provides additional information, such as, "There's a data check step missing here."

[1015] 6. The device records the audio again and analyzes it again using the speech recognition engine and emotion engine.

[1016] 7. The server generates an updated workflow diagram by adding new information and presents it adaptively based on the emotion data.

[1017] 8. The server automatically generates slides based on the diagram, providing information such as "common issues in data checking" and "solutions." The slide design is adjusted based on the emotion.

[1018] 9. Finally, the presentation slides and video are sent to the device for the user to review.

[1019] The above is a detailed description of the embodiment of this invention. This makes it possible to effectively implement BPR without specialized knowledge and promote in-house development of business processes. Furthermore, by combining it with an emotion engine, the user experience is improved, and more effective proposals and information presentation are realized.

[1020] The processing flow will be explained below.

[1021] Step 1:

[1022] The user verbally explains the workflow. For example, "I will explain the monthly closing process for the accounting department. First, we collect all transaction data..."

[1023] Step 2:

[1024] The terminal records the user's voice.

[1025] The recording module will be activated and the microphone will capture your voice.

[1026] Step 3:

[1027] The terminal transmits the recorded voice data to a voice recognition engine and an emotion engine.

[1028] Converts the recording into the appropriate format and sends the request to the engine API.

[1029] Step 4:

[1030] A voice recognition engine converts the voice data into text data.

[1031] The engine analyzes the audio waveform and generates text output.

[1032] Step 5:

[1033] The emotion engine recognizes the user's emotion from the voice data.

[1034] The emotion analysis module analyzes the speech features and outputs the emotional state.

[1035] Step 6:

[1036] A server receives the text data and emotion data obtained from the speech recognition engine and emotion engine.

[1037] The API response is received and the text data and emotion data are stored in the storage for analysis.

[1038] Step 7:

[1039] The server analyzes the text data and extracts the steps of the business flow.

[1040] The Natural Language Processing (NLP) module analyzes the text and extracts business flow steps.

[1041] Step 8:

[1042] The server generates an initial business flow diagram from the extracted steps.

[1043] Create flow diagrams using BPMN (Business Process Model and Notation) tools.

[1044] Step 9:

[1045] The server sends the generated business flow diagram to the terminal.

[1046] The flow diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[1047] Step 10:

[1048] The user checks the workflow diagram and verbally provides additional information or corrections if necessary.

[1049] Step 11:

[1050] The device records the audio again and sends it to the speech recognition engine and emotion engine.

[1051] The recording module will be activated again and the microphone will capture the audio.

[1052] The recording data is sent to the speech recognition engine and emotion engine.

[1053] Step 12:

[1054] The voice recognition engine converts the voice data into text data again.

[1055] The engine analyzes the audio waveform and generates text output.

[1056] Step 13:

[1057] The emotion engine again recognizes the user's emotion from the voice data.

[1058] The emotion analysis module analyzes the speech features and outputs the emotional state.

[1059] Step 14:

[1060] The server receives the reacquired text data and emotion data.

[1061] The API response is received and the text data and emotion data are stored in the storage for analysis.

[1062] Step 15:

[1063] The server updates the workflow diagram based on the information and emotion data acquired again.

[1064] New steps are incorporated into existing flow diagrams, and the diagrams are adaptively regenerated based on emotion data.

[1065] Step 16:

[1066] The server sends the updated business process diagram to the terminal.

[1067] The regenerated diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[1068] Step 17:

[1069] The server automatically analyzes common problems and their solutions based on the generated business flow diagram.

[1070] The issue extraction module analyzes the flow diagram and identifies potential issues from data patterns.

[1071] The solution suggestion module also takes into account the emotional data and suggests the best solution.

[1072] Step 18:

[1073] The terminal allows the user to confirm the presented problem and solution.

[1074] The problem and solution are presented on the screen, allowing the user to see it in the way that is most receptive to their emotions.

[1075] Step 19:

[1076] The user checks the issues and solutions and instructs corrections as necessary.

[1077] Step 20:

[1078] The server automatically generates presentation slides based on the business flow diagram, issues, and solutions.

[1079] The slide generation module embeds workflow diagrams, challenges, and solutions into templates, applying emotionally sensitive layouts and designs.

[1080] Step 21:

[1081] The server creates a video based on the generated slides and sends it to the terminal.

[1082] A video generation module animates the slides and provides narration.

[1083] Adjust the tone and speed of narration based on emotional data.

[1084] A video file is generated and sent to the device as an HTTP response.

[1085] Step 22:

[1086] Users can review the final slides and videos and download or share them as desired.

[1087] The above is a detailed description of the embodiment of this invention. This makes it possible to effectively implement BPR without specialized knowledge and promote in-house development of business processes. Furthermore, by combining it with an emotion engine, the user experience is improved, and more effective proposals and information presentation are realized.

[1088] Example 2

[1089] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1090] Conventional business process management systems make it difficult for users to create and update business flows concisely and quickly. Furthermore, they are unable to propose process improvements that take into account user emotions and immediate feedback, limiting the actual improvement of business efficiency. Furthermore, creating presentation materials and videos requires a great deal of effort and time, which means that insufficient resources can be allocated to essential business improvements.

[1091] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1092] In this invention, the server includes means for converting a workflow described orally by a user into text data using speech recognition technology, means for analyzing the text data to extract workflow steps and generate a workflow diagram, means for converting verbal corrections and supplementary information added by the user into text data using speech recognition technology again and updating the workflow diagram, means for automatically analyzing and presenting problems and their solutions based on the generated workflow diagram, means for automatically generating presentation slides based on the workflow diagram, problems, and solutions, means for creating videos based on the automatically generated slides, means for recording the user's explanation, means for analyzing and recording emotion data, and means for processing the user's voice data using a speech recognition engine and an emotion engine.This enables users to not only quickly and easily create and update workflows, but also enables adaptive suggestions based on emotion data and efficient creation of presentation materials and videos.

[1093] "Voice recognition technology" is a technology that analyzes a user's voice and converts it into text data.

[1094] "Text data" is a data format in which the user's voice is converted into text information.

[1095] A "business flow" is a visual representation of each step in a business process.

[1096] A "business flow diagram" is a graphical representation of the steps of a business flow extracted from text data.

[1097] "Emotion analysis" is a technology that recognizes and analyzes a user's emotional state from their voice data.

[1098] "Emotion data" is data that indicates the emotional state of the user obtained by emotion analysis.

[1099] "Presentation slides" are materials used to visually present business flow diagrams, challenges, and solutions.

[1100] "Video" refers to video content in the form of animated presentation slides with narration.

[1101] A "voice recognition engine" is a software module that analyzes voice waveforms and converts them into text data.

[1102] The "emotion engine" is a software module that analyzes the features of the user's voice data and recognizes their emotional state.

[1103] A "server" is a computer system that performs processes such as extracting steps in a business flow, generating business flow diagrams, analyzing and presenting problems and solutions, and generating slides and videos.

[1104] A "terminal" is a device that allows a user to input voice and check the generated business flow diagram, issues, and solutions.

[1105] This system converts a user's verbal description of a workflow into text data using speech recognition technology, analyzes the text data to extract workflow steps, and generates a workflow diagram. This system also has the ability to convert verbal corrections and supplementary information added by the user back into text data using speech recognition technology, updating the workflow diagram. Furthermore, it automatically analyzes and presents issues and solutions based on the generated workflow diagram. It also automatically generates presentation slides based on the workflow diagram, issues, and solutions, and creates videos based on those slides. Additionally, by incorporating an emotion engine that recognizes emotions from the user's voice, the system can adaptively present suggestions and information based on the user's emotional state.

[1106] Hardware and software used

[1107] This system uses the following hardware and software:

[1108] 1. Hardware:

[1109] Device: The device where the user inputs voice (e.g., smartphone, tablet, computer).

[1110] Microphone: An audio input device for capturing the user's voice.

[1111] Server: A computer system that processes and analyzes data.

[1112] 2. Software:

[1113] Speech recognition engine: A technology that converts speech waveforms into text, such as the Google Cloud Speech-to-Text API.

[1114] Emotion engine: Technology that analyzes emotions from voice data, such as IBM Watson Tone Analyzer.

[1115] NLP module: Natural language processing technology that analyzes text data and extracts business flow steps.

[1116] BPMN tool: A Business Process Model and Notation tool for generating business flow diagrams.

[1117] Slide generation module: Software that automatically generates slides for presentations, such as the Google Slides API.

[1118] Video Generation Module: Software that generates videos from slides, such as Adobe Premiere Pro API.

[1119] Specific examples of program processing

[1120] 1. The user verbally explains the workflow. For example, "I'll explain the monthly closing process for the accounting department. First, we collect all transaction data..."

[1121] 2. The device records the audio.

[1122] The recording module will be activated and the microphone will capture the audio.

[1123] 3. The device sends the recorded voice data to the speech recognition engine and emotion engine.

[1124] Converts the voice data into the appropriate format and sends requests to the speech recognition engine and emotion engine APIs.

[1125] 4. The speech recognition engine converts the voice data into text data.

[1126] Analyzes audio waveforms and generates text output.

[1127] 5. The emotion engine recognizes the user's emotions from the voice data.

[1128] Analyzes speech features and outputs emotional state.

[1129] 6. The server receives the text data and emotion data and stores them in the analysis storage.

[1130] 7. The server analyzes the text data and extracts the business flow steps.

[1131] The NLP module analyzes the text and extracts business flow steps.

[1132] 8. The server generates an initial business flow diagram from the extracted steps.

[1133] Create a flow diagram using a BPMN tool.

[1134] 9. The server sends the generated business flow diagram to the terminal.

[1135] Example prompt

[1136] "Please explain the monthly closing process for the accounting department."

[1137] "How do I add a missing step to my workflow diagram?"

[1138] "Based on this workflow, what challenges do you foresee?"

[1139] "Make a video based on the generated slides."

[1140] The above is an embodiment of the present invention. This allows even users without specialized knowledge to easily and quickly create and update workflows, thereby improving actual work efficiency. It also enables adaptive proposals that take into account the user's emotional data and the efficient creation of presentation materials and videos.

[1141] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1142] Step 1:

[1143] The user verbally explains the workflow.

[1144] Input: The user's verbal description. For example, "I'm going to explain the monthly closing process for the accounting department. First, I'll collect all the transaction data..."

[1145] Output: User's voice data.

[1146] Step 2:

[1147] The device records the user's voice.

[1148] Specific operation: The device's microphone captures the sound, and the recording module is activated to generate digital audio data.

[1149] Input: User's voice.

[1150] Output: Recorded audio data (WAV or MP3 format).

[1151] Step 3:

[1152] The device sends the recorded voice data to the voice recognition engine and emotion engine.

[1153] What happens: The recorded voice data is converted into an appropriate format and sent as a request to a speech recognition engine (e.g., Google Cloud Speech-to-Text API) and an emotion engine (e.g., IBM Watson Tone Analyzer API).

[1154] Input: Recorded audio data.

[1155] Output: The request data sent to the speech recognition engine and emotion engine.

[1156] Step 4:

[1157] A voice recognition engine converts the voice data into text data.

[1158] What it does: The speech recognition engine analyzes the speech waveform and generates corresponding text.

[1159] Input: Audio data.

[1160] Output: Text data (user's explanation).

[1161] Step 5:

[1162] The emotion engine recognizes the user's emotion from the voice data.

[1163] Specific operation: The emotion analysis module analyzes the voice features and outputs the user's emotional state (e.g., "excited" or "calm").

[1164] Input: Audio data.

[1165] Output: Emotion data (emotional state).

[1166] Step 6:

[1167] A server receives the text data and emotion data obtained from the speech recognition engine and emotion engine.

[1168] Specific operation: Receives the API response and stores the text data and emotion data in the analytical storage (database).

[1169] Input: Text data, emotion data.

[1170] Output: Data stored in analytical storage.

[1171] Step 7:

[1172] The server analyzes the text data and extracts the steps of the business flow.

[1173] Specific operation: The NLP module analyzes the text data and extracts each step of the business flow (e.g., "collect transaction data" and "verify data").

[1174] Input: Text data.

[1175] Output: A step in the business flow.

[1176] Step 8:

[1177] The server generates an initial business flow diagram from the extracted steps.

[1178] Specific operation: Create a business flow diagram using a BPMN tool and arrange the extracted steps as nodes and edges.

[1179] Input: A workflow step.

[1180] Output: Business flow diagram.

[1181] Step 9:

[1182] The server sends the generated business flow diagram to the terminal.

[1183] Specific operation: The business flow diagram is converted into an image (e.g. PNG) or XML format and sent to the terminal as an HTTP response.

[1184] Input: Business flow diagram.

[1185] Output: The workflow diagram sent to the terminal.

[1186] Step 10:

[1187] The user checks the workflow diagram and verbally provides additional information or corrections.

[1188] Input: The displayed business flow diagram, additional information or corrections made verbally by the user.

[1189] Output: Audio data including any user additions or corrections.

[1190] Step 11:

[1191] The device records the audio again and sends it to the speech recognition engine and emotion engine.

[1192] Specific operation: The recording module starts again, captures the user's voice, and generates digital voice data, which is then sent to the speech recognition engine and emotion engine.

[1193] Input: User spoken additions and corrections.

[1194] Output: Speech data sent to the speech recognition engine and emotion engine.

[1195] Step 12:

[1196] The voice recognition engine converts the voice data into text data again, which is received by the server.

[1197] Specific operation: The speech recognition engine analyzes the speech waveform and generates corresponding text. The generated text data is sent to the server, which receives it and stores it in a database.

[1198] Input: Audio data.

[1199] Output: Text data (additional information and corrections).

[1200] Step 13:

[1201] The emotion engine again recognizes the user's emotion from the voice data, and the data is received by the server.

[1202] Specific operation: The emotion analysis module analyzes the voice features and outputs a new emotional state. The generated emotional data is sent to the server and stored in a database.

[1203] Input: Audio data.

[1204] Output: Emotional data (emotional state in response to additional information or corrections).

[1205] Step 14:

[1206] The server updates the workflow diagram based on the information and emotion data acquired again.

[1207] Specific operation: Incorporate new steps (additional information or modifications) into an existing flow diagram, and change the diagram's color scheme and node display format based on emotion data.

[1208] Input: Text data of additional information and corrections, emotion data, and existing business flow diagram.

[1209] Output: An updated workflow diagram.

[1210] Step 15:

[1211] The server sends the updated business process diagram to the terminal.

[1212] Specific operation: The regenerated diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[1213] Input: Updated business flow diagram.

[1214] Output: The updated workflow diagram sent to the terminal.

[1215] Step 16:

[1216] The server automatically analyzes common problems and their solutions based on the generated business flow diagram.

[1217] Specific operation: The problem extraction module analyzes the business flow diagram and identifies potential problems (e.g., "insufficient data verification") from data patterns. The solution proposal module takes into account emotional data and proposes optimal solutions (e.g., "standardize data verification procedures").

[1218] Input: Updated workflow diagram, emotion data.

[1219] Output: Challenges and their solutions.

[1220] Step 17:

[1221] The terminal allows the user to confirm the presented problem and solution.

[1222] What it does: Display the problem and solution on the screen and allow the user to review it in the way that is most acceptable to them (e.g., read it out loud in a quiet voice).

[1223] Input: Challenges and solutions.

[1224] Output: The problem and solution presented to the user.

[1225] Step 18:

[1226] The user checks the issues and solutions and instructs corrections as necessary.

[1227] Input: The presented problem and solution.

[1228] Output: User feedback.

[1229] Step 19:

[1230] The server automatically generates presentation slides based on the business flow diagram, issues, and solutions.

[1231] How it works: The slide generation module uses Google Slides APIs and other tools to embed business flow diagrams, challenges, and solutions into templates, and applies emotionally-sensitive layouts and designs.

[1232] Input: Business flow diagram, problem, solution, sentiment data.

[1233] Output: Slides for presentation.

[1234] Step 20:

[1235] The server creates a video based on the generated slides and sends it to the terminal.

[1236] Specific operation: The video generation module uses APIs such as Adobe Premiere Pro to animate slides and add narration. It adjusts the tone and speed of the narration based on emotional data. The generated video file is sent to the device as an HTTP response.

[1237] Input: Presentation slides, emotion data.

[1238] Output: Video file.

[1239] Step 21:

[1240] Users can review the final slides and videos and download or share them as desired.

[1241] Specific behavior: The device displays the slides and video, and the user reviews them. If necessary, they are saved to local storage or shared with other users.

[1242] Input: Slides and videos.

[1243] Output: Verified slides and videos.

[1244] (Application example 2)

[1245] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1246] Conventional workflow creation systems require a great deal of effort, as they convert users' verbal descriptions into text data, then manually analyze that text data and generate a workflow diagram. Furthermore, if additional information or corrections are needed in the generated workflow diagram, they must be done manually again, which is inefficient. Furthermore, there was no function to automatically analyze and present issues and solutions based on the generated workflow diagram, so separate analysis work was required. Furthermore, conventional systems had difficulty recognizing users' emotions and providing adaptive information and suggestions, which prevented them from improving the user experience.

[1247] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1248] In this invention, the server includes: a means for converting a workflow described orally by a user into text data using speech recognition technology; a means for analyzing the text data to extract workflow steps and generate a workflow diagram; a means for converting verbal corrections and supplementary information added by the user into text data using speech recognition technology again to update the workflow diagram; a means for automatically analyzing and presenting problems and solutions based on the generated workflow diagram; a means for automatically generating a presentation document based on the workflow diagram, problems, and solutions; a means for creating visual materials based on the automatically generated document; a means for recognizing emotions from the user's voice; and a means for adaptively presenting information and making suggestions based on the emotion data. This enables efficient conversion of a workflow described orally by a user into text data and automatic generation and updating of a workflow diagram. Furthermore, the automatic presentation of problems and solutions improves the efficiency of the analysis process, and adaptive information presentation and suggestions based on the user's emotions significantly improves the user experience.

[1249] "Voice recognition technology" is a technology that converts voice into digital data and then into text data.

[1250] "Text data" refers to text information converted using voice recognition technology.

[1251] A "business flow" is a description of the steps or procedures of a particular business process.

[1252] A "workflow diagram" is a diagram that visually represents a workflow.

[1253] "Emotion recognition" is a technology that analyzes and recognizes a user's emotional state from their voice and other data.

[1254] A "means" is a specific method or device for achieving a specific purpose.

[1255] "User" refers to a person who uses the system.

[1256] An "issue" is a potential problem or obstacle in the workflow.

[1257] A "solution" is a method or means for addressing a problem.

[1258] A "presentation document" is a document that systematically summarizes business processes, issues, and solutions.

[1259] "Visual materials" are materials that are easy to understand visually and are created based on presentation documents.

[1260] "Adaptive information presentation" means providing information in an optimal form based on the user's emotional data.

[1261] MODE FOR CARRYING OUT THE INVENTION

[1262] This invention provides a system that converts a user's verbal description of a workflow into text data using speech recognition technology, analyzes the text data to extract workflow steps, and generates a workflow diagram. Furthermore, this system incorporates emotion recognition functionality, allowing it to make adaptive suggestions and present information based on the user's emotional state.

[1263] Speech and Emotion Recognition

[1264] 1. The user verbally explains the workflow. For example, "I'll explain the monthly closing process for the accounting department. First, we collect all transaction data..."

[1265] 2. The device records the user's voice using a microphone.

[1266] 3. The device sends the recorded voice data to a speech recognition engine and an emotion engine. The speech recognition engine uses Google's speech recognition API, and the emotion engine uses the Transformers sentiment-analysis pipeline.

[1267] 4. The speech recognition engine converts the speech data into text data, which is then generated.

[1268] 5. The emotion engine recognizes the user's emotions from the voice data and generates emotion data.

[1269] Business flow diagram generation and emotional adaptation

[1270] 6. The server analyzes the acquired text data and extracts the business flow steps using a Natural Language Processing (NLP) module.

[1271] 7. The server generates an initial business flow diagram from the extracted steps. The flow diagram is created using the business process model.

[1272] 8. The server sends the generated business flow diagram to the terminal. The diagram is sent as an image or in XML format.

[1273] Completion of missing information and emotional adaptation

[1274] 9. The user checks the workflow diagram and verbally provides additional information or corrections if necessary. For example, "There's a data check step missing here."

[1275] 10. The device records the audio again and sends it to the speech recognition engine and emotion engine.

[1276] 11. The voice recognition engine converts the voice data back into text data, which is received by the server.

[1277] 12. The server updates the workflow diagram based on the information and emotion data it has acquired again.

[1278] Presenting the problem and solution

[1279] 13. The server automatically analyzes and presents common problems and their solutions based on the generated business flow diagram.

[1280] 14. The device prompts the user to confirm the presented problem and solution.

[1281] Slide and video generation

[1282] 15. The server automatically generates presentation documents based on the business flow diagram, issues, and solutions. It uses a slide generation module.

[1283] 16. The server creates visual materials based on the generated documents and sends them to the device. The visual materials can be in the form of PowerPoint slides or videos.

[1284] Examples of concrete examples and prompts

[1285] As an example, a user might verbally describe:

[1286] "I will explain how to start and stop the production line. First, start the machine and then perform a line check. This is where the machine inspection step is required," he says.

[1287] Based on this prompt, the system will instruct the user as follows:

[1288] Please describe your workflow in the following format:

[1289] 1. First step explanation

[1290] 2. Next Steps Explained

[1291] 3. Next step explanation...

[1292] For example, 'First, start the machine, then do a line check...'"

[1293] As described above, the present invention provides a system that allows a user to explain a workflow and incorporates it efficiently into a workflow diagram using speech recognition and emotion recognition technology, thereby supporting problem solving and the creation of presentation materials.

[1294] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1295] Step 1:

[1296] The user verbally explains the workflow. The input is the workflow information in the form of the user's voice. For example, the user might say, "I'll explain the monthly closing process for the accounting department. First, I'll collect all the transaction data..." The output is the user's voice.

[1297] Step 2:

[1298] The device records the user's voice. The input is the user's voice. The device's microphone is used to capture the voice and store it as digital audio data. The output is the recorded audio data.

[1299] Step 3:

[1300] The device sends the recorded voice data to the speech recognition engine and emotion engine. The input is the recorded voice data. The voice data is converted into an appropriate format and a request is sent to the engine API. The output is an API request to the speech recognition engine and emotion engine.

[1301] Step 4:

[1302] A speech recognition engine converts speech data into text data. The input is speech data. The engine analyzes the speech waveform and generates text output. The output is text data.

[1303] Step 5:

[1304] The emotion engine recognizes the user's emotion from the voice data. The input is the voice data. The emotion analysis module analyzes the voice features and outputs the emotional state. The output is emotion data.

[1305] Step 6:

[1306] The server receives the text data and emotion data obtained from the speech recognition engine and emotion engine. The input is the text data and emotion data. The server receives the API response and stores the text data and emotion data in the storage for analysis. The output is the stored text data and emotion data.

[1307] Step 7:

[1308] The server analyzes text data and extracts workflow steps. The input is stored text data. The Natural Language Processing (NLP) module analyzes the text and extracts workflow steps. The output is the extracted workflow steps.

[1309] Step 8:

[1310] The server generates an initial business flow diagram from the extracted steps. The input is the extracted business flow steps. The flow diagram is created using the business process model. The output is the initial business flow diagram.

[1311] Step 9:

[1312] The server sends the generated business flow diagram to the terminal. The input is the initial business flow diagram. The flow diagram is converted into image or XML format and sent to the terminal as an HTTP response. The output is the business flow diagram sent to the terminal.

[1313] Step 10:

[1314] The user checks the workflow diagram and verbally provides additional information or corrections if necessary. The input is the workflow diagram and the user's voice. The user verbally provides supplementary information, for example, "There's a data check step missing here." The output is the user's voice, including the supplementary information.

[1315] Step 11:

[1316] The device records the audio again and sends it to the speech recognition engine and emotion engine. The input is the audio data including supplementary information. The recording module captures the audio and sends it to the speech recognition engine and emotion engine. The output is an API request to the speech recognition engine and emotion engine.

[1317] Step 12:

[1318] The speech recognition engine converts the speech data back into text data, which is received by the server. The input is speech data that includes supplemental information. The engine analyzes the speech waveform and generates text output, and the server receives the API response and stores it in the analysis storage. The output is text data that includes the stored supplemental information.

[1319] Step 13:

[1320] The emotion engine again recognizes the user's emotion from the voice data, which is received by the server. The input is voice data including supplementary information. The emotion analysis module analyzes the voice features and outputs the emotional state. The server receives the API response and stores it in the analysis storage. The output is the stored added emotion data.

[1321] Step 14:

[1322] The server updates the workflow diagram based on the newly acquired information and emotion data. The input is the new step and emotion data. The new step is incorporated into the existing workflow diagram, and the diagram is adaptively regenerated based on the emotion data. The output is the updated workflow diagram.

[1323] Step 15:

[1324] The server automatically generates a presentation document based on the generated business flow diagram, issues, and solutions. The input is the business flow diagram, issues, and solutions. The slide generation module embeds the business flow diagram, issues, and solutions in a template and applies an emotionally sensitive layout and design. The output is a presentation document.

[1325] Step 16:

[1326] The server creates visual materials based on the generated documents and sends them to the terminal. The input is the presentation document. The server animates the visual materials, adds narration, generates a video file, and sends it to the terminal as an HTTP response. The output is the visual materials.

[1327] The above is a detailed description of the specific process steps for carrying out the invention.

[1328] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1329] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1330] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1331] [Third embodiment]

[1332] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1333] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1334] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1335] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1336] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1337] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1338] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1339] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1340] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1341] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1342] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1343] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1344] This invention relates to a system that converts a user's verbal description of a workflow into text data using speech recognition technology, analyzes the text data to extract workflow steps, and generates a workflow diagram. This system also has the ability to convert verbal corrections and supplementary information added by the user back into text data using speech recognition technology, updating the workflow diagram. Furthermore, based on the generated workflow diagram, the system automatically analyzes and presents issues and their solutions. It also automatically generates presentation slides based on the workflow diagram, issues, and solutions, and creates videos based on those slides.

[1345] Explaining program processing in natural language

[1346] Voice Recognition

[1347] 1. The user verbally explains the workflow. For example, "I'd like to explain the monthly closing process for the accounting department. First, we collect all transaction data..."

[1348] 2. The device records the user's voice.

[1349] The recording module will be activated and the microphone will capture your voice.

[1350] 3. The device sends the recorded voice data to the voice recognition engine.

[1351] Converts the audio data into the appropriate format and sends the request to the Engine API.

[1352] 4. The voice recognition engine converts the voice data into text data.

[1353] The engine analyzes the audio waveform and generates text output.

[1354] 5. The server receives the text data obtained from the speech recognition engine.

[1355] The API response is received and the text data is stored in the analysis storage.

[1356] Generate a workflow diagram

[1357] 6. The server analyzes the text data obtained from the speech recognition engine.

[1358] The Natural Language Processing (NLP) module analyzes the text and extracts business flow steps.

[1359] 7. The server generates an initial business flow diagram from the extracted steps.

[1360] Create flow diagrams using BPMN (Business Process Model and Notation) tools.

[1361] 8. The server sends the generated business flow diagram to the terminal.

[1362] The flow diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[1363] Completion of missing information

[1364] 9. The user checks the workflow diagram and verbally provides additional information or corrections if necessary. For example, "There's a data check step missing here."

[1365] 10. The device records the audio again and sends it to the speech recognition engine.

[1366] The recording module will be activated again and the microphone will capture the audio.

[1367] The recording data is sent to the speech recognition engine.

[1368] 11. The speech recognition engine converts the speech back into text data, which is then received by the server.

[1369] As before, we analyze the audio waveform and generate text output.

[1370] Receive the API response and store it as additional information in the analytical storage.

[1371] 12. The server updates the business flow diagram based on the information obtained again.

[1372] Incorporate the new steps into an existing flow diagram and regenerate the updated diagram.

[1373] 13. The server sends the updated workflow diagram to the terminal.

[1374] The regenerated diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[1375] Presenting the problem and solution

[1376] 14. The server automatically analyzes common problems and their solutions based on the generated business flow diagram.

[1377] The issue extraction module analyzes the flow diagram and identifies potential issues from data patterns.

[1378] The solution proposal module searches for suitable solutions from a historical database and generates proposals.

[1379] 15. The terminal prompts the user to confirm the presented problem and solution.

[1380] The problem and solution are displayed on the screen so that the user can check it.

[1381] 16. The user reviews the problem and solution and suggests modifications if necessary. For example, "This solution doesn't apply here, so let's think of another way."

[1382] Slide and video generation

[1383] 17. The server automatically generates presentation slides based on the business flow diagram, issues, and solutions.

[1384] The slide generation module embeds business flow diagrams, challenges, and solutions into templates.

[1385] 18. The server creates a video based on the generated slides and sends it to the terminal.

[1386] A video generation module animates the slides and provides narration.

[1387] A video file is generated and sent to the device as an HTTP response.

[1388] 19. The user reviews the final slides and video and downloads or shares them as desired.

[1389] The device displays the slides and video, which the user can review.

[1390] Specific examples

[1391] 1. The user says, "I'd like to explain the monthly closing process for the accounting department. First, we'll collect all the transaction data..."

[1392] 2. The device records the audio and converts it into text using a speech recognition engine.

[1393] 3. The server extracts steps such as "collect transaction data" and generates an initial business flow diagram.

[1394] 4. The server sends the generated diagram to the terminal, where the user can confirm it.

[1395] 5. The user verbally provides additional information, such as, "There's a data check step missing here."

[1396] 6. The device will record the audio again and convert it into text using a speech recognition engine.

[1397] 7. The server generates an updated workflow diagram with the new information added.

[1398] 8. Based on the diagram, the server provides information such as "common issues in data checking" and "solutions," and automatically generates slides.

[1399] 9. Finally, the presentation slides and video are sent to the device for the user to review.

[1400] The above is a detailed description of the embodiment of the present invention, which enables effective BPR without specialized knowledge and promotes in-house development of business processes.

[1401] The processing flow will be explained below.

[1402] Step 1:

[1403] The user verbally explains the workflow. For example, "I will explain the monthly closing process for the accounting department. First, we collect all transaction data..."

[1404] Step 2:

[1405] The terminal records the user's voice.

[1406] The recording module will be activated and the microphone will capture your voice.

[1407] Step 3:

[1408] The device sends the recorded voice data to a voice recognition engine.

[1409] Converts the audio data into the appropriate format and sends the request to the Engine API.

[1410] Step 4:

[1411] A voice recognition engine converts the voice data into text data.

[1412] The engine analyzes the audio waveform and generates text output.

[1413] Step 5:

[1414] The server receives the text data obtained from the speech recognition engine.

[1415] The API response is received and the text data is stored in the analysis storage.

[1416] Step 6:

[1417] The server analyzes the text data and extracts the steps of the business flow.

[1418] The Natural Language Processing (NLP) module analyzes the text and extracts business flow steps.

[1419] Step 7:

[1420] The server generates an initial business flow diagram from the extracted steps.

[1421] Create flow diagrams using BPMN (Business Process Model and Notation) tools.

[1422] Step 8:

[1423] The server sends the generated business flow diagram to the terminal.

[1424] The flow diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[1425] Step 9:

[1426] The user checks the workflow diagram and verbally provides additional information or corrections if necessary. For example, "There's a data check step missing here."

[1427] Step 10:

[1428] The device will record the audio again and send it to the speech recognition engine.

[1429] The recording module will be activated again and the microphone will capture the audio.

[1430] The recording data is sent to the speech recognition engine.

[1431] Step 11:

[1432] The voice recognition engine converts the voice data into text data again, which is received by the server.

[1433] As before, we analyze the audio waveform and generate text output.

[1434] Receive the API response and store it as additional information in the analytical storage.

[1435] Step 12:

[1436] The server updates the business flow diagram based on the information retrieved again.

[1437] Incorporate the new steps into an existing flow diagram and regenerate the updated diagram.

[1438] Step 13:

[1439] The server sends the updated business process diagram to the terminal.

[1440] The regenerated diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[1441] Step 14:

[1442] The server automatically analyzes common problems and their solutions based on the generated business flow diagram.

[1443] The issue extraction module analyzes the flow diagram and identifies potential issues from data patterns.

[1444] The solution proposal module searches for suitable solutions from a historical database and generates proposals.

[1445] Step 15:

[1446] The terminal allows the user to confirm the presented problem and solution.

[1447] The problem and solution are displayed on the screen so that the user can check it.

[1448] Step 16:

[1449] The user reviews the problem and solution and suggests modifications if necessary, for example, "This solution doesn't apply here, so let's think of another way."

[1450] Step 17:

[1451] The server automatically generates presentation slides based on the business flow diagram, issues, and solutions.

[1452] The slide generation module embeds business flow diagrams, challenges, and solutions into templates.

[1453] Step 18:

[1454] The server creates a video based on the generated slides and sends it to the terminal.

[1455] A video generation module animates the slides and provides narration.

[1456] A video file is generated and sent to the device as an HTTP response.

[1457] Step 19:

[1458] Users can review the final slides and videos and download or share them as desired.

[1459] The device presents slides and videos, which the user reviews.

[1460] The above is a detailed description of the specific processing steps of the program.

[1461] Example 1

[1462] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1463] Traditionally, the process of formulating workflows was largely manual, requiring time and effort. Furthermore, reviewing and improving workflows required extensive specialized knowledge and skills, making the process inefficient. Furthermore, updating workflow diagrams, identifying issues, and proposing solutions were complex, creating a need for automation.

[1464] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1465] In this invention, the server includes means for converting a business process described orally by a user into text data using speech recognition technology, means for analyzing the text data to extract business process steps and generate a business process diagram, means for converting verbal corrections and supplementary information added by the user into text data again using speech recognition technology and updating the business process diagram, means for automatically analyzing and presenting issues and their solutions based on the generated business process diagram, means for automatically generating presentation slides based on the business process diagram, issues, and solutions, and means for creating videos based on the automatically generated slides. This enables effective improvement of business processes without specialized knowledge and promotes in-house development of business processes.

[1466] A "user" is someone who uses the system to explain the workflow and add corrections or supplementary information.

[1467] "Speech recognition technology" is a technology that analyzes the voice data of a user's oral explanation and converts it into text data.

[1468] "Text data" refers to the text information of voice data converted using voice recognition technology.

[1469] "Analysis" refers to the process of classifying text data, interpreting its meaning and structure, and extracting business flow steps.

[1470] A "business flow step" refers to an individual task or processing stage in a business process.

[1471] A "business flow diagram" is a diagram that visually represents the steps of a business flow.

[1472] An "issue" is a problem or obstacle that occurs in a business flow.

[1473] A "solution" is a method or means for solving a particular problem.

[1474] "Presentation slides" are materials used to visually present information such as business flow diagrams, issues, and solutions.

[1475] "Video" refers to dynamic video content created by adding animation and narration to automatically generated slides.

[1476] This invention relates to a system that uses speech recognition technology to generate a workflow diagram based on a user's verbal description of the workflow, thereby improving the efficiency of business processes. Specifically, the system converts the speech data into text data, analyzes it, extracts the workflow steps, and automatically generates a workflow diagram. Furthermore, the system is characterized by the fact that the workflow diagram is constantly updated with the latest information as the user verbally provides additional information or corrections, and automatically presents issues and their solutions.

[1477] To implement this system, the following hardware and software are required:

[1478] Device: A device used to record audio and communicate with the server. Examples include smartphones, tablets, and computers.

[1479] Server: The central computer system that provides the speech recognition and text analysis technologies. This may include a cloud server or a dedicated server.

[1480] Speech recognition engine: Software that converts a user's voice into text data. Representative engines include Google Cloud Speech-to-Text API and IBM Watson Speech to Text API.

[1481] Natural Language Processing (NLP) module: A software module for analyzing text data and extracting business flow steps.

[1482] BPMN (Business Process Model and Notation) tool: A dedicated tool for generating business flow diagrams.

[1483] Data analysis module: Software for automatically analyzing issues and their solutions from business flow diagrams.

[1484] Slide generation module: A module for automatically generating slides for presentations.

[1485] Video Generation Module: A module for creating videos based on generated slides.

[1486] The specific operating procedure of the system is as follows: first, the user verbally explains the workflow, which is then recorded by the device. The recorded voice data is sent to the server, where it is converted into text data by a speech recognition engine. Next, the server analyzes the text data using an NLP module and extracts the workflow steps. A BPMN tool generates a workflow diagram based on the extracted steps.

[1487] If the user verbally adds or modifies information to the diagram, the device records the voice again and sends it to a speech recognition engine to convert it into text data. The server updates the workflow diagram based on the new information and automatically analyzes and presents the issues and solutions. The server then uses a slide generation module to create presentation slides based on the workflow diagram, issues, and solutions. Furthermore, the server uses a video generation module to create a video based on the created slides and sends it to the device.

[1488] Specific examples

[1489] The user verbally explains, "I'd like to explain the monthly closing process for the accounting department. First, we'll collect all transaction data..." The device records the speech and converts it into text data using a voice recognition engine. The server extracts steps such as "collect transaction data" and generates an initial business flow diagram. The server sends the diagram it has generated to the device, where the user can confirm it.

[1490] The user verbally provides additional information, saying, "There's a data check step missing here." The device records the voice again and converts it into text data using a speech recognition engine. The server adds the new information and generates an updated business flow diagram. Based on the diagram, the server provides "common issues in data checks" and "solutions," and automatically generates slides. Finally, the presentation slides and video are sent to the device, where the user can review them.

[1491] The above is a detailed description of the embodiment of the present invention. This system enables effective improvement of business processes without specialized knowledge, and promotes automation and in-house production of business flows.

[1492] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1493] Step 1:

[1494] The user verbally explains the workflow. For example, "I'll explain the monthly closing process for the accounting department. First, we'll collect all transaction data..." This explanation becomes the initial data input.

[1495] Step 2:

[1496] The terminal records the user's voice. The recording module is activated and the device's microphone captures the voice. This captured voice data becomes the input data for the next step.

[1497] Step 3:

[1498] The device sends the recorded voice data to the speech recognition engine. The voice data is converted to an appropriate format (e.g., WAV, FLAC) and sent to the speech recognition engine using an HTTP request. This makes the voice data the target data for conversion.

[1499] Step 4:

[1500] The speech recognition engine converts the speech data into text data. It analyzes the speech waveform and generates text output. For example, the text data generated is "Collecting transaction data." This text data becomes the input data for analysis.

[1501] Step 5:

[1502] The server receives the text data obtained from the speech recognition engine. The JSON-formatted text data is received as an API response and stored in the analysis storage. This stored data becomes the input data for subsequent analysis processing.

[1503] Step 6:

[1504] The server analyzes the text data obtained from the speech recognition engine. The NLP module analyzes the text data, extracts nouns and verbs, and identifies the steps in the business flow. For example, steps such as "collect transaction data" and "verify data" are extracted. The results of this analysis become the input data for generating the flow diagram.

[1505] Step 7:

[1506] The server extracts the business flow steps from the analysis results. The NLP module lists the steps obtained and clarifies the order and relationships required to generate a business flow diagram. This becomes the preparatory data for generating the flow diagram.

[1507] Step 8:

[1508] The server generates a business flow diagram based on the extracted steps. Using a BPMN tool, a business flow diagram is created that visually represents a series of steps. For example, a flow diagram of "Collect transaction data" → "Confirm data" → "Complete monthly closing process" is generated. This generated flow diagram becomes the input data for the next import.

[1509] Step 9:

[1510] The server sends the generated business flow diagram to the terminal. The flow diagram is converted into an image or XML format and sent to the terminal as an HTTP response. This allows the user to check the specific flow diagram.

[1511] Step 10:

[1512] The user checks the workflow diagram and verbally provides additional information or corrections if necessary. For example, they might say, "There's a data check step missing here." This additional information becomes the input data for the next step.

[1513] Step 11:

[1514] The device records the audio again and sends it to the speech recognition engine. The recording module is then activated again, capturing additional audio, converting it to the appropriate format, and sending it to the engine, which then uses it as input data for reconversion.

[1515] Step 12:

[1516] The speech recognition engine converts the speech into text data again. It analyzes the additional speech and generates text containing "data check." This text data becomes the input data for analysis.

[1517] Step 13:

[1518] The server receives and analyzes the new text data. It analyzes the updated text data and recognizes the new step "Data Check." This updated data becomes the updated input data for the flow diagram.

[1519] Step 14:

[1520] The server generates an updated business flow diagram. The new steps are incorporated into the existing flow diagram, and the flow diagram is regenerated in the format of "Collect transaction data" → "Data check" → "Data confirmation" → "Monthly closing process completed." This flow diagram will be the updated data.

[1521] Step 15:

[1522] The server sends the updated business flow diagram to the terminal. The regenerated flow diagram is converted back to PNG or SVG format and sent to the terminal. The user can then review it.

[1523] Step 16:

[1524] The server analyzes the issues and solutions based on the workflow diagram. The data analysis module analyzes the workflow diagram and identifies issues at specific steps. For example, an issue such as "data checking takes a long time" is extracted. This issue and solution become the input data for the next generated slide.

[1525] Step 17:

[1526] The terminal presents the problem and the solution to the user. The problem and the solution received from the server are displayed on the screen so that the user can confirm them. This allows the user to decide on the next action.

[1527] Step 18:

[1528] The user checks the problem and solution and gives instructions for corrections as necessary. For example, the user may give verbal instructions such as, "It would be better to use the current tool, so please think of another solution." This additional instruction becomes prompt data for the next step.

[1529] Step 19:

[1530] The server generates presentation slides based on the business flow diagram, issues, and solutions. The slide generation module embeds the business flow diagram, issues, and solutions into a template and generates a slide-format file. The output is a PowerPoint or PDF.

[1531] Step 20:

[1532] The server generates a video based on the slides and sends it to the device. The video generation module animates the generated slides, synthesizes the narration from the text, and creates a video file. This video becomes the final output data for the user.

[1533] Step 21:

[1534] The user reviews the final slides and video, and downloads or shares them as needed. The device displays the slides and video, allowing the user to review them and proceed to the next step in the process.

[1535] (Application example 1)

[1536] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1537] Optimizing business processes in factories is important from the perspective of improving efficiency and productivity on-site, but conventional methods require specialized knowledge and require a great deal of time and effort. In particular, it has been difficult to grasp the real-time progress of work within a factory and immediately present optimization proposals. In addition, organizing business flows and creating presentation materials also require time and effort, which leads to a decline in work efficiency. For this reason, there is a need for a system that can efficiently manage business processes within a factory and make immediate proposals.

[1538] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1539] In this invention, the server includes: means for converting a work flow verbally explained by a user into text data using voice recognition technology; means for analyzing the text data to extract workflow steps and generate a workflow diagram; means for converting verbal corrections and supplementary information added by the user into text data using voice recognition technology again to update the workflow diagram; means for automatically analyzing and presenting problems and solutions based on the generated workflow diagram; means for automatically generating presentation images based on the workflow diagram, problems, and solutions; means for creating videos based on the automatically generated images; means for using a robot equipped with a voice recognition module that allows a user to verbally explain factory operations and work instructions in real time; and means for analyzing voice data recorded by the robot and providing optimization suggestions based on the workflow diagram. This enables efficient management of factory business processes and real-time optimization suggestions on-site.

[1540] "User" refers to a person who uses the system to verbally explain the workflow and provide corrections and supplementary information.

[1541] "Verbal explanation" refers to the act of using voice to communicate the flow of work and correction information.

[1542] "Voice recognition technology" refers to the technology that analyzes voice and converts it into text data.

[1543] "Text data" refers to text data converted from speech using speech recognition technology.

[1544] "Workflow" refers to a series of steps or procedures for accomplishing a particular task or work.

[1545] A "step" refers to an individual unit of work within a workflow.

[1546] A "workflow diagram" refers to a visual representation of a workflow.

[1547] A "problem" refers to a glitch or inefficiency that occurs within a workflow.

[1548] A "solution" refers to a specific method or proposal for solving a problem.

[1549] "Presentation images" refers to diagrams and presentation slides that visually illustrate workflows, business problems, and solutions.

[1550] "Video creation" refers to the process of creating visual explanations in video format based on presentation images.

[1551] A "robot equipped with a voice recognition module" refers to a robot that has the ability to record voice data and analyze the voice in real time.

[1552] "Optimization proposals based on workflow diagrams" refers to proposals that analyze workflow diagrams to identify problems and show how to improve them.

[1553] "Real-time" refers to processing and providing information immediately, without delay.

[1554] This invention is a system for efficiently managing work processes in factories and making optimization proposals in real time. This system uses voice recognition technology to convert a user's verbal explanation into text data, analyzes that data to generate a workflow diagram, and finally creates images and videos for presentations based on the proposals.

[1555] System Overview

[1556] The system includes the following elements:

[1557] 1. Robot equipped with a voice recognition module

[1558] It records factory operations and work instructions in real time and converts user voice into text data.

[1559] Specific hardware: robot, microphone

[1560] Software used: Google Speech-to-Text API

[1561] 2. Server

[1562] Analyzes character data and extracts workflow steps.

[1563] Generate workflow diagrams to analyze and present problems and their solutions.

[1564] Automatically generate images and videos for presentations.

[1565] Software used: nltk, pydotplus, PaddleOCR

[1566] System action

[1567] Voice Recognition

[1568] The robot embedded in the device records work instructions and explanations in real time within the factory and converts them into text data using the Google Speech-to-Text API, which the device then sends to a server.

[1569] Text data analysis

[1570] The server receives the text data and uses natural language processing tools such as nltk to extract each step of the workflow. Pydotplus is used to generate a workflow diagram from the extracted steps.

[1571] Presenting the problem and solution

[1572] Based on the generated workflow diagram, the server automatically analyzes potential problems, searches for appropriate solutions from past data, and presents them to the user. This analysis utilizes data pattern mining technology.

[1573] Image and video generation for presentations

[1574] The server generates presentation images and explanatory videos based on the generated workflow diagram and analysis results. The presentation images are generated by embedding information in templates, and the explanatory videos are generated by animating the images and adding narration.

[1575] Specific examples

[1576] 1. The user says, "I will explain the assembly procedure for the production line. First, prepare all the parts, then connect part A to part B. After that, perform a quality check..."

[1577] 2. The device records the audio and converts it into text using the Google Speech-to-Text API.

[1578] 3. The server receives the text data and extracts steps such as "preparing parts," "connecting parts A and B," and "quality check."

[1579] 4. The server generates a workflow diagram based on the extracted information and displays it on the terminal for the user to review.

[1580] 5. The user verbally adds additional information, saying, "After the quality check, there will be packaging."

[1581] 6. The device records the audio again and converts it into text data using the Google Speech-to-Text API.

[1582] 7. The server takes in the new information and generates an updated workflow diagram.

[1583] 8. Based on this workflow diagram and analysis results, the server provides optimization suggestions and generates presentation images and explanatory videos.

[1584] Prompt Sentence Examples

[1585] "Please explain in detail the workflow of your factory's production line. Then, please add any missing steps or areas for improvement. Also, please add details of each process."

[1586] This system allows users to effectively optimize factory operations without specialized knowledge, making it possible to improve the efficiency of work flows.

[1587] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1588] Step 1:

[1589] The terminal records the work instructions and workflows that users verbally explain in the factory. Specifically, a microphone installed on the robot captures the user's voice, and the recording module converts the voice into a digital format. The recording data is generated and sent to the server. The output based on the input is digital voice data.

[1590] Step 2:

[1591] The server sends the received digital voice data to the Google Speech-to-Text API, which converts the voice into text data (character data). The server then calls the API of the speech recognition engine and converts the voice input into text format. The input data is voice data, and the output data is character data.

[1592] Step 3:

[1593] The server analyzes the text data and extracts each step of the workflow. Specifically, the Natural Language Processing (NLP) module analyzes the text data and identifies a series of work steps from the content. The input is the text data, and the output is a list of the analyzed work steps.

[1594] Step 4:

[1595] The server generates a workflow diagram based on the extracted work steps. The server uses pydotplus to create a flow diagram that connects each step. The input is a list of work steps, and the output is a workflow diagram. The flow diagram is generated in image or XML format and sent to the terminal.

[1596] Step 5:

[1597] The user checks the workflow diagram and provides additional verbal explanations if there is missing information or if corrections are required. Specifically, when the user explains again, the device re-records the voice and sends it to the server. The input data is the user's voice, and the output data is voice data including supplementary information.

[1598] Step 6:

[1599] The server again converts the audio data of the additional explanation into text data using the Google Speech-to-Text API and analyzes the text data as before. The NLP module analyzes the added text information and updates the existing workflow diagram. The input is the audio data containing the supplementary information, and the output is the updated text data.

[1600] Step 7:

[1601] The server regenerates the workflow diagram based on the updated work steps. It uses pydotplus to incorporate the new steps into the existing diagram and recreates the updated diagram. The input is the updated list of work steps, and the output is the updated workflow diagram. The updated diagram is also sent to the terminal.

[1602] Step 8:

[1603] The server automatically analyzes and presents potential problems and their solutions based on the workflow diagram. Using a problem extraction module and a solution presentation module, the server analyzes the generated workflow diagram, identifies problems, and searches a database for appropriate solutions. The input is the workflow diagram, and the output is a list of problems and solutions, which is also sent to the terminal.

[1604] Step 9:

[1605] The server automatically generates images and videos for presentations based on the workflow diagram, problems, and solutions. The server uses a slide generation module and a video generation module to embed information in templates and create slides and videos. The input is the workflow diagram, problems, and a list of solutions, and the output is the images and video files for presentations. Finally, these are also sent to the terminal for the user to review.

[1606] Each step is explained in detail to help you understand how the system streamlines factory operations in real time.

[1607] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1608] This invention relates to a system that converts a user's verbal description of a workflow into text data using speech recognition technology, analyzes the text data to extract workflow steps, and generates a workflow diagram. This system also has the ability to convert verbal corrections and supplementary information added by the user back into text data using speech recognition technology, updating the workflow diagram. Based on the generated workflow diagram, the system automatically analyzes and presents issues and their solutions. Furthermore, the system automatically generates presentation slides based on the workflow diagram, issues, and solutions, and creates videos based on those slides.

[1609] Furthermore, by incorporating an emotion engine that recognizes emotions from the user's voice, the present invention realizes adaptive suggestions and information presentation based on the user's emotional state. This emotion engine also simultaneously acquires and stores user emotion data, which can be used for later analysis and improvement.

[1610] Explaining program processing in natural language

[1611] Speech and Emotion Recognition

[1612] 1. The user verbally explains the workflow. For example, "I'll explain the monthly closing process for the accounting department. First, we collect all transaction data..."

[1613] 2. The device records the user's voice.

[1614] The recording module will be activated and the microphone will capture your voice.

[1615] 3. The device sends the recorded voice data to the speech recognition engine and emotion engine.

[1616] Converts the audio data into the appropriate format and sends the request to the Engine API.

[1617] 4. The speech recognition engine converts the voice data into text data.

[1618] The engine analyzes the audio waveform and generates text output.

[1619] 5. The emotion engine recognizes the user's emotions from the voice data.

[1620] The emotion analysis module analyzes the speech features and outputs the emotional state.

[1621] 6. The server receives the text data and emotion data obtained from the speech recognition engine and emotion engine.

[1622] The API response is received and the text data and emotion data are stored in the storage for analysis.

[1623] Business flow diagram generation and emotional adaptation

[1624] 6. The server analyzes the text data and extracts the steps of the business flow.

[1625] The Natural Language Processing (NLP) module analyzes the text and extracts business flow steps.

[1626] 7. The server generates an initial business flow diagram from the extracted steps.

[1627] Create flow diagrams using BPMN (Business Process Model and Notation) tools.

[1628] 8. The server sends the generated business flow diagram to the terminal.

[1629] The flow diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[1630] Completion of missing information and emotional adaptation

[1631] 9. The user checks the workflow diagram and verbally provides additional information or corrections if necessary. For example, "There's a data check step missing here."

[1632] 10. The device records the audio again and sends it to the speech recognition engine and emotion engine.

[1633] The recording module will be activated again and the microphone will capture the audio.

[1634] The recording data is sent to the speech recognition engine and emotion engine.

[1635] 11. The voice recognition engine converts the voice data back into text data, which is received by the server.

[1636] As before, we analyze the audio waveform and generate text output.

[1637] Receive the API response and store it as additional information in the analytical storage.

[1638] 12. The emotion engine again recognizes the user's emotion from the voice data, and the server receives it.

[1639] The emotion analysis module analyzes the speech features and outputs the emotional state.

[1640] The API response is received and the added emotion data is also stored in the storage for analysis.

[1641] 13. The server updates the workflow diagram based on the information and emotion data obtained again.

[1642] New steps are incorporated into existing flow diagrams, and the diagrams are adaptively regenerated based on emotion data.

[1643] 14. The server sends the updated business flow diagram to the terminal.

[1644] The regenerated diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[1645] Presenting the problem and solution

[1646] 15. The server automatically analyzes common problems and their solutions based on the generated business flow diagram.

[1647] The issue extraction module analyzes the flow diagram and identifies potential issues from data patterns.

[1648] The solution suggestion module also takes into account the emotional data and suggests the best solution.

[1649] 16. The device prompts the user to confirm the presented problem and solution.

[1650] The problem and solution are presented on the screen, allowing the user to see it in the way that is most receptive to their emotions.

[1651] 17. The user reviews the problem and solution and suggests modifications if necessary. For example, "This solution doesn't work here, so let's think of another way."

[1652] Slide and video generation

[1653] 18. The server automatically generates presentation slides based on the business flow diagram, issues, and solutions.

[1654] The slide generation module embeds workflow diagrams, challenges, and solutions into templates, applying emotionally sensitive layouts and designs.

[1655] 19. The server creates a video based on the generated slides and sends it to the terminal.

[1656] A video generation module animates the slides and provides narration.

[1657] Adjust the tone and speed of narration based on emotional data.

[1658] A video file is generated and sent to the device as an HTTP response.

[1659] 20. User reviews the final slides and video and downloads or shares as desired.

[1660] The device displays slides and videos, which the user reviews.

[1661] Specific examples

[1662] 1. The user says, "I'd like to explain the monthly closing process for the accounting department. First, we'll collect all the transaction data..."

[1663] 2. The device records the audio and analyzes it using a voice recognition engine and emotion engine.

[1664] 3. The server extracts steps such as "collect transaction data" and generates an initial workflow diagram. If the user's emotion is recognized as "excited," the information is presented in a calm tone.

[1665] 4. The server sends the generated diagram to the terminal, where the user can confirm it.

[1666] 5. The user verbally provides additional information, such as, "There's a data check step missing here."

[1667] 6. The device records the audio again and analyzes it again using the speech recognition engine and emotion engine.

[1668] 7. The server generates an updated workflow diagram by adding new information and presents it adaptively based on the emotion data.

[1669] 8. The server automatically generates slides based on the diagram, providing information such as "common issues in data checking" and "solutions." The slide design is adjusted based on the emotion.

[1670] 9. Finally, the presentation slides and video are sent to the device for the user to review.

[1671] The above is a detailed description of the embodiment of this invention. This makes it possible to effectively implement BPR without specialized knowledge and promote in-house development of business processes. Furthermore, by combining it with an emotion engine, the user experience is improved, and more effective proposals and information presentation are realized.

[1672] The processing flow will be explained below.

[1673] Step 1:

[1674] The user verbally explains the workflow. For example, "I will explain the monthly closing process for the accounting department. First, we collect all transaction data..."

[1675] Step 2:

[1676] The terminal records the user's voice.

[1677] The recording module will be activated and the microphone will capture your voice.

[1678] Step 3:

[1679] The terminal transmits the recorded voice data to a voice recognition engine and an emotion engine.

[1680] Converts the recording into the appropriate format and sends the request to the engine API.

[1681] Step 4:

[1682] A voice recognition engine converts the voice data into text data.

[1683] The engine analyzes the audio waveform and generates text output.

[1684] Step 5:

[1685] The emotion engine recognizes the user's emotion from the voice data.

[1686] The emotion analysis module analyzes the speech features and outputs the emotional state.

[1687] Step 6:

[1688] A server receives the text data and emotion data obtained from the speech recognition engine and emotion engine.

[1689] The API response is received and the text data and emotion data are stored in the storage for analysis.

[1690] Step 7:

[1691] The server analyzes the text data and extracts the steps of the business flow.

[1692] The Natural Language Processing (NLP) module analyzes the text and extracts business flow steps.

[1693] Step 8:

[1694] The server generates an initial business flow diagram from the extracted steps.

[1695] Create flow diagrams using BPMN (Business Process Model and Notation) tools.

[1696] Step 9:

[1697] The server sends the generated business flow diagram to the terminal.

[1698] The flow diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[1699] Step 10:

[1700] The user checks the workflow diagram and verbally provides additional information or corrections if necessary.

[1701] Step 11:

[1702] The device records the audio again and sends it to the speech recognition engine and emotion engine.

[1703] The recording module will be activated again and the microphone will capture the audio.

[1704] The recording data is sent to the speech recognition engine and emotion engine.

[1705] Step 12:

[1706] The voice recognition engine converts the voice data into text data again.

[1707] The engine analyzes the audio waveform and generates text output.

[1708] Step 13:

[1709] The emotion engine again recognizes the user's emotion from the voice data.

[1710] The emotion analysis module analyzes the speech features and outputs the emotional state.

[1711] Step 14:

[1712] The server receives the reacquired text data and emotion data.

[1713] The API response is received and the text data and emotion data are stored in the storage for analysis.

[1714] Step 15:

[1715] The server updates the workflow diagram based on the information and emotion data acquired again.

[1716] New steps are incorporated into existing flow diagrams, and the diagrams are adaptively regenerated based on emotion data.

[1717] Step 16:

[1718] The server sends the updated business process diagram to the terminal.

[1719] The regenerated diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[1720] Step 17:

[1721] The server automatically analyzes common problems and their solutions based on the generated business flow diagram.

[1722] The issue extraction module analyzes the flow diagram and identifies potential issues from data patterns.

[1723] The solution suggestion module also takes into account the emotional data and suggests the best solution.

[1724] Step 18:

[1725] The terminal allows the user to confirm the presented problem and solution.

[1726] The problem and solution are presented on the screen, allowing the user to see it in the way that is most receptive to their emotions.

[1727] Step 19:

[1728] The user checks the issues and solutions and instructs corrections as necessary.

[1729] Step 20:

[1730] The server automatically generates presentation slides based on the business flow diagram, issues, and solutions.

[1731] The slide generation module embeds workflow diagrams, challenges, and solutions into templates, applying emotionally sensitive layouts and designs.

[1732] Step 21:

[1733] The server creates a video based on the generated slides and sends it to the terminal.

[1734] A video generation module animates the slides and provides narration.

[1735] Adjust the tone and speed of narration based on emotional data.

[1736] A video file is generated and sent to the device as an HTTP response.

[1737] Step 22:

[1738] Users can review the final slides and videos and download or share them as desired.

[1739] The above is a detailed description of the embodiment of this invention. This makes it possible to effectively implement BPR without specialized knowledge and promote in-house development of business processes. Furthermore, by combining it with an emotion engine, the user experience is improved, and more effective proposals and information presentation are realized.

[1740] Example 2

[1741] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1742] Conventional business process management systems make it difficult for users to create and update business flows concisely and quickly. Furthermore, they are unable to propose process improvements that take into account user emotions and immediate feedback, limiting the actual improvement of business efficiency. Furthermore, creating presentation materials and videos requires a great deal of effort and time, which means that insufficient resources can be allocated to essential business improvements.

[1743] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1744] In this invention, the server includes means for converting a workflow described orally by a user into text data using speech recognition technology, means for analyzing the text data to extract workflow steps and generate a workflow diagram, means for converting verbal corrections and supplementary information added by the user into text data using speech recognition technology again and updating the workflow diagram, means for automatically analyzing and presenting problems and their solutions based on the generated workflow diagram, means for automatically generating presentation slides based on the workflow diagram, problems, and solutions, means for creating videos based on the automatically generated slides, means for recording the user's explanation, means for analyzing and recording emotion data, and means for processing the user's voice data using a speech recognition engine and an emotion engine.This enables users to not only quickly and easily create and update workflows, but also enables adaptive suggestions based on emotion data and efficient creation of presentation materials and videos.

[1745] "Voice recognition technology" is a technology that analyzes a user's voice and converts it into text data.

[1746] "Text data" is a data format in which the user's voice is converted into text information.

[1747] A "business flow" is a visual representation of each step in a business process.

[1748] A "business flow diagram" is a graphical representation of the steps of a business flow extracted from text data.

[1749] "Emotion analysis" is a technology that recognizes and analyzes a user's emotional state from their voice data.

[1750] "Emotion data" is data that indicates the emotional state of the user obtained by emotion analysis.

[1751] "Presentation slides" are materials used to visually present business flow diagrams, challenges, and solutions.

[1752] "Video" refers to video content in the form of animated presentation slides with narration.

[1753] A "voice recognition engine" is a software module that analyzes voice waveforms and converts them into text data.

[1754] The "emotion engine" is a software module that analyzes the features of the user's voice data and recognizes their emotional state.

[1755] A "server" is a computer system that performs processes such as extracting steps in a business flow, generating business flow diagrams, analyzing and presenting problems and solutions, and generating slides and videos.

[1756] A "terminal" is a device that allows a user to input voice and check the generated business flow diagram, issues, and solutions.

[1757] This system converts a user's verbal description of a workflow into text data using speech recognition technology, analyzes the text data to extract workflow steps, and generates a workflow diagram. This system also has the ability to convert verbal corrections and supplementary information added by the user back into text data using speech recognition technology, updating the workflow diagram. Furthermore, it automatically analyzes and presents issues and solutions based on the generated workflow diagram. It also automatically generates presentation slides based on the workflow diagram, issues, and solutions, and creates videos based on those slides. Additionally, by incorporating an emotion engine that recognizes emotions from the user's voice, the system can adaptively present suggestions and information based on the user's emotional state.

[1758] Hardware and software used

[1759] This system uses the following hardware and software:

[1760] 1. Hardware:

[1761] Device: The device where the user inputs voice (e.g., smartphone, tablet, computer).

[1762] Microphone: An audio input device for capturing the user's voice.

[1763] Server: A computer system that processes and analyzes data.

[1764] 2. Software:

[1765] Speech recognition engine: A technology that converts speech waveforms into text, such as the Google Cloud Speech-to-Text API.

[1766] Emotion engine: Technology that analyzes emotions from voice data, such as IBM Watson Tone Analyzer.

[1767] NLP module: Natural language processing technology that analyzes text data and extracts business flow steps.

[1768] BPMN tool: A Business Process Model and Notation tool for generating business flow diagrams.

[1769] Slide generation module: Software that automatically generates slides for presentations, such as the Google Slides API.

[1770] Video Generation Module: Software that generates videos from slides, such as Adobe Premiere Pro API.

[1771] Specific examples of program processing

[1772] 1. The user verbally explains the workflow. For example, "I'll explain the monthly closing process for the accounting department. First, we collect all transaction data..."

[1773] 2. The device records the audio.

[1774] The recording module will be activated and the microphone will capture the audio.

[1775] 3. The device sends the recorded voice data to the speech recognition engine and emotion engine.

[1776] Converts the voice data into the appropriate format and sends requests to the speech recognition engine and emotion engine APIs.

[1777] 4. The speech recognition engine converts the voice data into text data.

[1778] Analyzes audio waveforms and generates text output.

[1779] 5. The emotion engine recognizes the user's emotions from the voice data.

[1780] Analyzes speech features and outputs emotional state.

[1781] 6. The server receives the text data and emotion data and stores them in the analysis storage.

[1782] 7. The server analyzes the text data and extracts the business flow steps.

[1783] The NLP module analyzes the text and extracts business flow steps.

[1784] 8. The server generates an initial business flow diagram from the extracted steps.

[1785] Create a flow diagram using a BPMN tool.

[1786] 9. The server sends the generated business flow diagram to the terminal.

[1787] Example prompt

[1788] "Please explain the monthly closing process for the accounting department."

[1789] "How do I add a missing step to my workflow diagram?"

[1790] "Based on this workflow, what challenges do you foresee?"

[1791] "Make a video based on the generated slides."

[1792] The above is an embodiment of the present invention. This allows even users without specialized knowledge to easily and quickly create and update workflows, thereby improving actual work efficiency. It also enables adaptive proposals that take into account the user's emotional data and the efficient creation of presentation materials and videos.

[1793] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1794] Step 1:

[1795] The user verbally explains the workflow.

[1796] Input: The user's verbal description. For example, "I'm going to explain the monthly closing process for the accounting department. First, I'll collect all the transaction data..."

[1797] Output: User's voice data.

[1798] Step 2:

[1799] The device records the user's voice.

[1800] Specific operation: The device's microphone captures the sound, and the recording module is activated to generate digital audio data.

[1801] Input: User's voice.

[1802] Output: Recorded audio data (WAV or MP3 format).

[1803] Step 3:

[1804] The device sends the recorded voice data to the voice recognition engine and emotion engine.

[1805] What happens: The recorded voice data is converted into an appropriate format and sent as a request to a speech recognition engine (e.g., Google Cloud Speech-to-Text API) and an emotion engine (e.g., IBM Watson Tone Analyzer API).

[1806] Input: Recorded audio data.

[1807] Output: The request data sent to the speech recognition engine and emotion engine.

[1808] Step 4:

[1809] A voice recognition engine converts the voice data into text data.

[1810] What it does: The speech recognition engine analyzes the speech waveform and generates corresponding text.

[1811] Input: Audio data.

[1812] Output: Text data (user's explanation).

[1813] Step 5:

[1814] The emotion engine recognizes the user's emotion from the voice data.

[1815] Specific operation: The emotion analysis module analyzes the voice features and outputs the user's emotional state (e.g., "excited" or "calm").

[1816] Input: Audio data.

[1817] Output: Emotion data (emotional state).

[1818] Step 6:

[1819] A server receives the text data and emotion data obtained from the speech recognition engine and emotion engine.

[1820] Specific operation: Receives the API response and stores the text data and emotion data in the analytical storage (database).

[1821] Input: Text data, emotion data.

[1822] Output: Data stored in analytical storage.

[1823] Step 7:

[1824] The server analyzes the text data and extracts the steps of the business flow.

[1825] Specific operation: The NLP module analyzes the text data and extracts each step of the business flow (e.g., "collect transaction data" and "verify data").

[1826] Input: Text data.

[1827] Output: A step in the business flow.

[1828] Step 8:

[1829] The server generates an initial business flow diagram from the extracted steps.

[1830] Specific operation: Create a business flow diagram using a BPMN tool and arrange the extracted steps as nodes and edges.

[1831] Input: A workflow step.

[1832] Output: Business flow diagram.

[1833] Step 9:

[1834] The server sends the generated business flow diagram to the terminal.

[1835] Specific operation: The business flow diagram is converted into an image (e.g. PNG) or XML format and sent to the terminal as an HTTP response.

[1836] Input: Business flow diagram.

[1837] Output: The workflow diagram sent to the terminal.

[1838] Step 10:

[1839] The user checks the workflow diagram and verbally provides additional information or corrections.

[1840] Input: The displayed business flow diagram, additional information or corrections made verbally by the user.

[1841] Output: Audio data including any user additions or corrections.

[1842] Step 11:

[1843] The device records the audio again and sends it to the speech recognition engine and emotion engine.

[1844] Specific operation: The recording module starts again, captures the user's voice, and generates digital voice data, which is then sent to the speech recognition engine and emotion engine.

[1845] Input: User spoken additions and corrections.

[1846] Output: Speech data sent to the speech recognition engine and emotion engine.

[1847] Step 12:

[1848] The voice recognition engine converts the voice data into text data again, which is received by the server.

[1849] Specific operation: The speech recognition engine analyzes the speech waveform and generates corresponding text. The generated text data is sent to the server, which receives it and stores it in a database.

[1850] Input: Audio data.

[1851] Output: Text data (additional information and corrections).

[1852] Step 13:

[1853] The emotion engine again recognizes the user's emotion from the voice data, and the data is received by the server.

[1854] Specific operation: The emotion analysis module analyzes the voice features and outputs a new emotional state. The generated emotional data is sent to the server and stored in a database.

[1855] Input: Audio data.

[1856] Output: Emotional data (emotional state in response to additional information or corrections).

[1857] Step 14:

[1858] The server updates the workflow diagram based on the information and emotion data acquired again.

[1859] Specific operation: Incorporate new steps (additional information or modifications) into an existing flow diagram, and change the diagram's color scheme and node display format based on emotion data.

[1860] Input: Text data of additional information and corrections, emotion data, and existing business flow diagram.

[1861] Output: An updated workflow diagram.

[1862] Step 15:

[1863] The server sends the updated business process diagram to the terminal.

[1864] Specific operation: The regenerated diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[1865] Input: Updated business flow diagram.

[1866] Output: The updated workflow diagram sent to the terminal.

[1867] Step 16:

[1868] The server automatically analyzes common problems and their solutions based on the generated business flow diagram.

[1869] Specific operation: The problem extraction module analyzes the business flow diagram and identifies potential problems (e.g., "insufficient data verification") from data patterns. The solution proposal module takes into account emotional data and proposes optimal solutions (e.g., "standardize data verification procedures").

[1870] Input: Updated workflow diagram, emotion data.

[1871] Output: Challenges and their solutions.

[1872] Step 17:

[1873] The terminal allows the user to confirm the presented problem and solution.

[1874] What it does: Display the problem and solution on the screen and allow the user to review it in the way that is most acceptable to them (e.g., read it out loud in a quiet voice).

[1875] Input: Challenges and solutions.

[1876] Output: The problem and solution presented to the user.

[1877] Step 18:

[1878] The user checks the issues and solutions and instructs corrections as necessary.

[1879] Input: The presented problem and solution.

[1880] Output: User feedback.

[1881] Step 19:

[1882] The server automatically generates presentation slides based on the business flow diagram, issues, and solutions.

[1883] How it works: The slide generation module uses Google Slides APIs and other tools to embed business flow diagrams, challenges, and solutions into templates, and applies emotionally-sensitive layouts and designs.

[1884] Input: Business flow diagram, problem, solution, sentiment data.

[1885] Output: Slides for presentation.

[1886] Step 20:

[1887] The server creates a video based on the generated slides and sends it to the terminal.

[1888] Specific operation: The video generation module uses APIs such as Adobe Premiere Pro to animate slides and add narration. It adjusts the tone and speed of the narration based on emotional data. The generated video file is sent to the device as an HTTP response.

[1889] Input: Presentation slides, emotion data.

[1890] Output: Video file.

[1891] Step 21:

[1892] Users can review the final slides and videos and download or share them as desired.

[1893] Specific behavior: The device displays the slides and video, and the user reviews them. If necessary, they are saved to local storage or shared with other users.

[1894] Input: Slides and videos.

[1895] Output: Verified slides and videos.

[1896] (Application example 2)

[1897] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1898] Conventional workflow creation systems require a great deal of effort, as they convert users' verbal descriptions into text data, then manually analyze that text data and generate a workflow diagram. Furthermore, if additional information or corrections are needed in the generated workflow diagram, they must be done manually again, which is inefficient. Furthermore, there was no function to automatically analyze and present issues and solutions based on the generated workflow diagram, so separate analysis work was required. Furthermore, conventional systems had difficulty recognizing users' emotions and providing adaptive information and suggestions, which prevented them from improving the user experience.

[1899] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1900] In this invention, the server includes: a means for converting a workflow described orally by a user into text data using speech recognition technology; a means for analyzing the text data to extract workflow steps and generate a workflow diagram; a means for converting verbal corrections and supplementary information added by the user into text data using speech recognition technology again to update the workflow diagram; a means for automatically analyzing and presenting problems and solutions based on the generated workflow diagram; a means for automatically generating a presentation document based on the workflow diagram, problems, and solutions; a means for creating visual materials based on the automatically generated document; a means for recognizing emotions from the user's voice; and a means for adaptively presenting information and making suggestions based on the emotion data. This enables efficient conversion of a workflow described orally by a user into text data and automatic generation and updating of a workflow diagram. Furthermore, the automatic presentation of problems and solutions improves the efficiency of the analysis process, and adaptive information presentation and suggestions based on the user's emotions significantly improves the user experience.

[1901] "Voice recognition technology" is a technology that converts voice into digital data and then into text data.

[1902] "Text data" refers to text information converted using voice recognition technology.

[1903] A "business flow" is a description of the steps or procedures of a particular business process.

[1904] A "workflow diagram" is a diagram that visually represents a workflow.

[1905] "Emotion recognition" is a technology that analyzes and recognizes a user's emotional state from their voice and other data.

[1906] A "means" is a specific method or device for achieving a specific purpose.

[1907] "User" refers to a person who uses the system.

[1908] An "issue" is a potential problem or obstacle in the workflow.

[1909] A "solution" is a method or means for addressing a problem.

[1910] A "presentation document" is a document that systematically summarizes business processes, issues, and solutions.

[1911] "Visual materials" are materials that are easy to understand visually and are created based on presentation documents.

[1912] "Adaptive information presentation" means providing information in an optimal form based on the user's emotional data.

[1913] MODE FOR CARRYING OUT THE INVENTION

[1914] This invention provides a system that converts a user's verbal description of a workflow into text data using speech recognition technology, analyzes the text data to extract workflow steps, and generates a workflow diagram. Furthermore, this system incorporates emotion recognition functionality, allowing it to make adaptive suggestions and present information based on the user's emotional state.

[1915] Speech and Emotion Recognition

[1916] 1. The user verbally explains the workflow. For example, "I'll explain the monthly closing process for the accounting department. First, we collect all transaction data..."

[1917] 2. The device records the user's voice using a microphone.

[1918] 3. The device sends the recorded voice data to a speech recognition engine and an emotion engine. The speech recognition engine uses Google's speech recognition API, and the emotion engine uses the Transformers sentiment-analysis pipeline.

[1919] 4. The speech recognition engine converts the speech data into text data, which is then generated.

[1920] 5. The emotion engine recognizes the user's emotions from the voice data and generates emotion data.

[1921] Business flow diagram generation and emotional adaptation

[1922] 6. The server analyzes the acquired text data and extracts the business flow steps using a Natural Language Processing (NLP) module.

[1923] 7. The server generates an initial business flow diagram from the extracted steps. The flow diagram is created using the business process model.

[1924] 8. The server sends the generated business flow diagram to the terminal. The diagram is sent as an image or in XML format.

[1925] Completion of missing information and emotional adaptation

[1926] 9. The user checks the workflow diagram and verbally provides additional information or corrections if necessary. For example, "There's a data check step missing here."

[1927] 10. The device records the audio again and sends it to the speech recognition engine and emotion engine.

[1928] 11. The voice recognition engine converts the voice data back into text data, which is received by the server.

[1929] 12. The server updates the workflow diagram based on the information and emotion data it has acquired again.

[1930] Presenting the problem and solution

[1931] 13. The server automatically analyzes and presents common problems and their solutions based on the generated business flow diagram.

[1932] 14. The device prompts the user to confirm the presented problem and solution.

[1933] Slide and video generation

[1934] 15. The server automatically generates presentation documents based on the business flow diagram, issues, and solutions. It uses a slide generation module.

[1935] 16. The server creates visual materials based on the generated documents and sends them to the device. The visual materials can be in the form of PowerPoint slides or videos.

[1936] Examples of concrete examples and prompts

[1937] As an example, a user might verbally describe:

[1938] "I will explain how to start and stop the production line. First, start the machine and then perform a line check. This is where the machine inspection step is required," he says.

[1939] Based on this prompt, the system will instruct the user as follows:

[1940] Please describe your workflow in the following format:

[1941] 1. First step explanation

[1942] 2. Next Steps Explained

[1943] 3. Next step explanation...

[1944] For example, 'First, start the machine, then do a line check...'"

[1945] As described above, the present invention provides a system that allows a user to explain a workflow and incorporates it efficiently into a workflow diagram using speech recognition and emotion recognition technology, thereby supporting problem solving and the creation of presentation materials.

[1946] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1947] Step 1:

[1948] The user verbally explains the workflow. The input is the workflow information in the form of the user's voice. For example, the user might say, "I'll explain the monthly closing process for the accounting department. First, I'll collect all the transaction data..." The output is the user's voice.

[1949] Step 2:

[1950] The device records the user's voice. The input is the user's voice. The device's microphone is used to capture the voice and store it as digital audio data. The output is the recorded audio data.

[1951] Step 3:

[1952] The device sends the recorded voice data to the speech recognition engine and emotion engine. The input is the recorded voice data. The voice data is converted into an appropriate format and a request is sent to the engine API. The output is an API request to the speech recognition engine and emotion engine.

[1953] Step 4:

[1954] A speech recognition engine converts speech data into text data. The input is speech data. The engine analyzes the speech waveform and generates text output. The output is text data.

[1955] Step 5:

[1956] The emotion engine recognizes the user's emotion from the voice data. The input is the voice data. The emotion analysis module analyzes the voice features and outputs the emotional state. The output is emotion data.

[1957] Step 6:

[1958] The server receives the text data and emotion data obtained from the speech recognition engine and emotion engine. The input is the text data and emotion data. The server receives the API response and stores the text data and emotion data in the storage for analysis. The output is the stored text data and emotion data.

[1959] Step 7:

[1960] The server analyzes text data and extracts workflow steps. The input is stored text data. The Natural Language Processing (NLP) module analyzes the text and extracts workflow steps. The output is the extracted workflow steps.

[1961] Step 8:

[1962] The server generates an initial business flow diagram from the extracted steps. The input is the extracted business flow steps. The flow diagram is created using the business process model. The output is the initial business flow diagram.

[1963] Step 9:

[1964] The server sends the generated business flow diagram to the terminal. The input is the initial business flow diagram. The flow diagram is converted into image or XML format and sent to the terminal as an HTTP response. The output is the business flow diagram sent to the terminal.

[1965] Step 10:

[1966] The user checks the workflow diagram and verbally provides additional information or corrections if necessary. The input is the workflow diagram and the user's voice. The user verbally provides supplementary information, for example, "There's a data check step missing here." The output is the user's voice, including the supplementary information.

[1967] Step 11:

[1968] The device records the audio again and sends it to the speech recognition engine and emotion engine. The input is the audio data including supplementary information. The recording module captures the audio and sends it to the speech recognition engine and emotion engine. The output is an API request to the speech recognition engine and emotion engine.

[1969] Step 12:

[1970] The speech recognition engine converts the speech data back into text data, which is received by the server. The input is speech data that includes supplemental information. The engine analyzes the speech waveform and generates text output, and the server receives the API response and stores it in the analysis storage. The output is text data that includes the stored supplemental information.

[1971] Step 13:

[1972] The emotion engine again recognizes the user's emotion from the voice data, which is received by the server. The input is voice data including supplementary information. The emotion analysis module analyzes the voice features and outputs the emotional state. The server receives the API response and stores it in the analysis storage. The output is the stored added emotion data.

[1973] Step 14:

[1974] The server updates the workflow diagram based on the newly acquired information and emotion data. The input is the new step and emotion data. The new step is incorporated into the existing workflow diagram, and the diagram is adaptively regenerated based on the emotion data. The output is the updated workflow diagram.

[1975] Step 15:

[1976] The server automatically generates a presentation document based on the generated business flow diagram, issues, and solutions. The input is the business flow diagram, issues, and solutions. The slide generation module embeds the business flow diagram, issues, and solutions in a template and applies an emotionally sensitive layout and design. The output is a presentation document.

[1977] Step 16:

[1978] The server creates visual materials based on the generated documents and sends them to the terminal. The input is the presentation document. The server animates the visual materials, adds narration, generates a video file, and sends it to the terminal as an HTTP response. The output is the visual materials.

[1979] The above is a detailed description of the specific process steps for carrying out the invention.

[1980] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1981] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1982] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1983] [Fourth embodiment]

[1984] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1985] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1986] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1987] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1988] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1989] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1990] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1991] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1992] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1993] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1994] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1995] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1996] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1997] This invention relates to a system that converts a user's verbal description of a workflow into text data using speech recognition technology, analyzes the text data to extract workflow steps, and generates a workflow diagram. This system also has the ability to convert verbal corrections and supplementary information added by the user back into text data using speech recognition technology, updating the workflow diagram. Furthermore, based on the generated workflow diagram, the system automatically analyzes and presents issues and their solutions. It also automatically generates presentation slides based on the workflow diagram, issues, and solutions, and creates videos based on those slides.

[1998] Explaining program processing in natural language

[1999] Voice Recognition

[2000] 1. The user verbally explains the workflow. For example, "I'd like to explain the monthly closing process for the accounting department. First, we collect all transaction data..."

[2001] 2. The device records the user's voice.

[2002] The recording module will be activated and the microphone will capture your voice.

[2003] 3. The device sends the recorded voice data to the voice recognition engine.

[2004] Converts the audio data into the appropriate format and sends the request to the Engine API.

[2005] 4. The voice recognition engine converts the voice data into text data.

[2006] The engine analyzes the audio waveform and generates text output.

[2007] 5. The server receives the text data obtained from the speech recognition engine.

[2008] The API response is received and the text data is stored in the analysis storage.

[2009] Generate a workflow diagram

[2010] 6. The server analyzes the text data obtained from the speech recognition engine.

[2011] The Natural Language Processing (NLP) module analyzes the text and extracts business flow steps.

[2012] 7. The server generates an initial business flow diagram from the extracted steps.

[2013] Create flow diagrams using BPMN (Business Process Model and Notation) tools.

[2014] 8. The server sends the generated business flow diagram to the terminal.

[2015] The flow diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[2016] Completion of missing information

[2017] 9. The user checks the workflow diagram and verbally provides additional information or corrections if necessary. For example, "There's a data check step missing here."

[2018] 10. The device records the audio again and sends it to the speech recognition engine.

[2019] The recording module will be activated again and the microphone will capture the audio.

[2020] The recording data is sent to the speech recognition engine.

[2021] 11. The speech recognition engine converts the speech back into text data, which is then received by the server.

[2022] As before, we analyze the audio waveform and generate text output.

[2023] Receive the API response and store it as additional information in the analytical storage.

[2024] 12. The server updates the business flow diagram based on the information obtained again.

[2025] Incorporate the new steps into an existing flow diagram and regenerate the updated diagram.

[2026] 13. The server sends the updated workflow diagram to the terminal.

[2027] The regenerated diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[2028] Presenting the problem and solution

[2029] 14. The server automatically analyzes common problems and their solutions based on the generated business flow diagram.

[2030] The issue extraction module analyzes the flow diagram and identifies potential issues from data patterns.

[2031] The solution proposal module searches for suitable solutions from a historical database and generates proposals.

[2032] 15. The terminal prompts the user to confirm the presented problem and solution.

[2033] The problem and solution are displayed on the screen so that the user can check it.

[2034] 16. The user reviews the problem and solution and suggests modifications if necessary. For example, "This solution doesn't apply here, so let's think of another way."

[2035] Slide and video generation

[2036] 17. The server automatically generates presentation slides based on the business flow diagram, issues, and solutions.

[2037] The slide generation module embeds business flow diagrams, challenges, and solutions into templates.

[2038] 18. The server creates a video based on the generated slides and sends it to the terminal.

[2039] A video generation module animates the slides and provides narration.

[2040] A video file is generated and sent to the device as an HTTP response.

[2041] 19. The user reviews the final slides and video and downloads or shares them as desired.

[2042] The device displays the slides and video, which the user can review.

[2043] Specific examples

[2044] 1. The user says, "I'd like to explain the monthly closing process for the accounting department. First, we'll collect all the transaction data..."

[2045] 2. The device records the audio and converts it into text using a speech recognition engine.

[2046] 3. The server extracts steps such as "collect transaction data" and generates an initial business flow diagram.

[2047] 4. The server sends the generated diagram to the terminal, where the user can confirm it.

[2048] 5. The user verbally provides additional information, such as, "There's a data check step missing here."

[2049] 6. The device will record the audio again and convert it into text using a speech recognition engine.

[2050] 7. The server generates an updated workflow diagram with the new information added.

[2051] 8. Based on the diagram, the server provides information such as "common issues in data checking" and "solutions," and automatically generates slides.

[2052] 9. Finally, the presentation slides and video are sent to the device for the user to review.

[2053] The above is a detailed description of the embodiment of the present invention, which enables effective BPR without specialized knowledge and promotes in-house development of business processes.

[2054] The processing flow will be explained below.

[2055] Step 1:

[2056] The user verbally explains the workflow. For example, "I will explain the monthly closing process for the accounting department. First, we collect all transaction data..."

[2057] Step 2:

[2058] The terminal records the user's voice.

[2059] The recording module will be activated and the microphone will capture your voice.

[2060] Step 3:

[2061] The device sends the recorded voice data to a voice recognition engine.

[2062] Converts the audio data into the appropriate format and sends the request to the Engine API.

[2063] Step 4:

[2064] A voice recognition engine converts the voice data into text data.

[2065] The engine analyzes the audio waveform and generates text output.

[2066] Step 5:

[2067] The server receives the text data obtained from the speech recognition engine.

[2068] The API response is received and the text data is stored in the analysis storage.

[2069] Step 6:

[2070] The server analyzes the text data and extracts the steps of the business flow.

[2071] The Natural Language Processing (NLP) module analyzes the text and extracts business flow steps.

[2072] Step 7:

[2073] The server generates an initial business flow diagram from the extracted steps.

[2074] Create flow diagrams using BPMN (Business Process Model and Notation) tools.

[2075] Step 8:

[2076] The server sends the generated business flow diagram to the terminal.

[2077] The flow diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[2078] Step 9:

[2079] The user checks the workflow diagram and verbally provides additional information or corrections if necessary. For example, "There's a data check step missing here."

[2080] Step 10:

[2081] The device will record the audio again and send it to the speech recognition engine.

[2082] The recording module will be activated again and the microphone will capture the audio.

[2083] The recording data is sent to the speech recognition engine.

[2084] Step 11:

[2085] The voice recognition engine converts the voice data into text data again, which is received by the server.

[2086] As before, we analyze the audio waveform and generate text output.

[2087] Receive the API response and store it as additional information in the analytical storage.

[2088] Step 12:

[2089] The server updates the business flow diagram based on the information retrieved again.

[2090] Incorporate the new steps into an existing flow diagram and regenerate the updated diagram.

[2091] Step 13:

[2092] The server sends the updated business process diagram to the terminal.

[2093] The regenerated diagram is converted into an image or XML format and sent to the terminal as an HTTP response.

[2094] Step 14:

[2095] The server automatically analyzes common problems and their solutions based on the generated business flow diagram.

[2096] The issue extraction module analyzes the flow diagram and identifies potential issues from data patterns.

[2097] The solution proposal module searches for suitable solutions from a historical database and generates proposals.

[2098] Step 15:

[2099] The terminal allows the user to confirm the presented problem and solution.

[2100] The problem and solution are displayed on the screen so that the user can check it.

[2101] Step 16:

[2102] The user reviews the problem and solution and suggests modifications if necessary, for example, "This solution doesn't apply here, so let's think of another way."

[2103] Step 17:

[2104] The server automatically generates presentation slides based on the business flow diagram, issues, and solutions.

[2105] The slide generation module embeds business flow diagrams, challenges, and solutions into templates.

[2106] Step 18:

[2107] The server creates a video based on the generated slides and sends it to the terminal.

[2108] A video generation module animates the slides and provides narration.

[2109] A video file is generated and sent to the device as an HTTP response.

[2110] Step 19:

[2111] Users can review the final slides and videos and download or share them as desired.

[2112] The device presents slides and videos, which the user reviews.

[2113] The above is a detailed description of the specific processing steps of the program.

[2114] Example 1

[2115] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2116] Traditionally, the process of formulating workflows was largely manual, requiring time and effort. Furthermore, reviewing and improving workflows required extensive specialized knowledge and skills, making the process inefficient. Furthermore, updating workflow diagrams, identifying issues, and proposing solutions were complex, creating a need for automation.

[2117] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2118] In this invention, the server includes means for converting a business process described orally by a user into text data using speech recognition technology, means for analyzing the text data to extract business process steps and generate a business process diagram, means for converting verbal corrections and supplementary information added by the user into text data again using speech recognition technology and updating the business process diagram, means for automatically analyzing and presenting issues and their solutions based on the generated business process diagram, means for automatically generating presentation slides based on the business process diagram, issues, and solutions, and means for creating videos based on the automatically generated slides. This enables effective improvement of business processes without specialized knowledge and promotes in-house development of business processes.

[2119] A "user" is someone who uses the system to explain the workflow and add corrections or supplementary information.

[2120] "Speech recognition technology" is a technology that analyzes the voice data of a user's oral explanation and converts it into text data.

[2121] "Text data" refers to the text information of voice data converted using voice recognition technology.

[2122] "Analysis" refers to the process of classifying text data, interpreting its meaning and structure, and extracting business flow steps.

[2123] A "business flow step" refers to an individual task or processing stage in a business process.

[2124] A "business flow diagram" is a diagram that visually represents the steps of a business flow.

[2125] An "issue" is a problem or obstacle that occurs in a business flow.

[2126] A "solution" is a method or means for solving a particular problem.

[2127] "Presentation slides" are materials used to visually present information such as business flow diagrams, issues, and solutions.

[2128] "Video" refers to dynamic video content created by adding animation and narration to automatically generated slides.

[2129] This invention relates to a system that uses speech recognition technology to generate a workflow diagram based on a user's verbal description of the workflow, thereby improving the efficiency of business processes. Specifically, the system converts the speech data into text data, analyzes it, extracts the workflow steps, and automatically generates a workflow diagram. Furthermore, the system is characterized by the fact that the workflow diagram is constantly updated with the latest information as the user verbally provides additional information or corrections, and automatically presents issues and their solutions.

[2130] To implement this system, the following hardware and software are required:

[2131] Device: A device used to record audio and communicate with the server. Examples include smartphones, tablets, and computers.

[2132] Server: The central computer system that provides the speech recognition and text analysis technologies. This may include a cloud server or a dedicated server.

[2133] Speech recognition engine: Software that converts a user's voice into text data. Representative engines include Google Cloud Speech-to-Text API and IBM Watson Speech to Text API.

[2134] Natural Language Processing (NLP) module: A software module for analyzing text data and extracting business flow steps.

[2135] BPMN (Business Process Model and Notation) tool: A dedicated tool for generating business flow diagrams.

[2136] Data analysis module: Software for automatically analyzing issues and their solutions from business flow diagrams.

[2137] Slide generation module: A module for automatically generating slides for presentations.

[2138] Video Generation Module: A module for creating videos based on generated slides.

[2139] The specific operating procedure of the system is as follows: first, the user verbally explains the workflow, which is then recorded by the device. The recorded voice data is sent to the server, where it is converted into text data by a speech recognition engine. Next, the server analyzes the text data using an NLP module and extracts the workflow steps. A BPMN tool generates a workflow diagram based on the extracted steps.

[2140] If the user verbally adds or modifies information to the diagram, the device records the voice again and sends it to a speech recognition engine to convert it into text data. The server updates the workflow diagram based on the new information and automatically analyzes and presents the issues and solutions. The server then uses a slide generation module to create presentation slides based on the workflow diagram, issues, and solutions. Furthermore, the server uses a video generation module to create a video based on the created slides and sends it to the device.

[2141] Specific examples

[2142] The user verbally explains, "I'd like to explain the monthly closing process for the accounting department. First, we'll collect all transaction data..." The device records the speech and converts it into text data using a voice recognition engine. The server extracts steps such as "collect transaction data" and generates an initial business flow diagram. The server sends the diagram it has generated to the device, where the user can confirm it.

[2143] The user verbally provides additional information, saying, "There's a data check step missing here." The device records the voice again and converts it into text data using a speech recognition engine. The server adds the new information and generates an updated business flow diagram. Based on the diagram, the server provides "common issues in data checks" and "solutions," and automatically generates slides. Finally, the presentation slides and video are sent to the device, where the user can review them.

[2144] The above is a detailed description of the embodiment of the present invention. This system enables effective improvement of business processes without specialized knowledge, and promotes automation and in-house production of business flows.

[2145] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2146] Step 1:

[2147] The user verbally explains the workflow. For example, "I'll explain the monthly closing process for the accounting department. First, we'll collect all transaction data..." This explanation becomes the initial data input.

[2148] Step 2:

[2149] The terminal records the user's voice. The recording module is activated and the device's microphone captures the voice. This captured voice data becomes the input data for the next step.

[2150] Step 3:

[2151] The device sends the recorded voice data to the speech recognition engine. The voice data is converted to an appropriate format (e.g., WAV, FLAC) and sent to the speech recognition engine using an HTTP request. This makes the voice data the target data for conversion.

[2152] Step 4:

[2153] The speech recognition engine converts the speech data into text data. It analyzes the speech waveform and generates text output. For example, the text data generated is "Collecting transaction data." This text data becomes the input data for analysis.

[2154] Step 5:

[2155] The server receives the text data obtained from the speech recognition engine. The JSON-formatted text data is received as an API response and stored in the analysis storage. This stored data becomes the input data for subsequent analysis processing.

[2156] Step 6:

[2157] The server analyzes the text data obtained from the speech recognition engine. The NLP module analyzes the text data, extracts nouns and verbs, and identifies the steps in the business flow. For example, steps such as "collect transaction data" and "verify data" are extracted. The results of this analysis become the input data for generating the flow diagram.

[2158] Step 7:

[2159] The server extracts the business flow steps from the analysis results. The NLP module lists the steps obtained and clarifies the order and relationships required to generate a business flow diagram. This becomes the preparatory data for generating the flow diagram.

[2160] Step 8:

[2161] The server generates a business flow diagram based on the extracted steps. Using a BPMN tool, a business flow diagram is created that visually represents a series of steps. For example, a flow diagram of "Collect transaction data" → "Confirm data" → "Complete monthly closing process" is generated. This generated flow diagram becomes the input data for the next import.

[2162] Step 9:

[2163] The server sends the generated business flow diagram to the terminal. The flow diagram is converted into an image or XML format and sent to the terminal as an HTTP response. This allows the user to check the specific flow diagram.

[2164] Step 10:

[2165] The user checks the workflow diagram and verbally provides additional information or corrections if necessary. For example, they might say, "There's a data check step missing here." This additional information becomes the input data for the next step.

[2166] Step 11:

[2167] The device records the audio again and sends it to the speech recognition engine. The recording module is then activated again, capturing additional audio, converting it to the appropriate format, and sending it to the engine, which then uses it as input data for reconversion.

[2168] Step 12:

[2169] The speech recognition engine converts the speech into text data again. It analyzes the additional speech and generates text containing "data check." This text data becomes the input data for analysis.

[2170] Step 13:

[2171] The server receives and analyzes the new text data. It analyzes the updated text data and recognizes the new step "Data Check." This updated data becomes the updated input data for the flow diagram.

[2172] Step 14:

[2173] The server generates an updated business flow diagram. The new steps are incorporated into the existing flow diagram, and the flow diagram is regenerated in the format of "Collect transaction data" → "Data check" → "Data confirmation" → "Monthly closing process completed." This flow diagram will be the updated data.

[2174] Step 15:

[2175] The server sends the updated business flow diagram to the terminal. The regenerated flow diagram is converted back to PNG or SVG format and sent to the terminal. The user can then review it.

[2176] Step 16:

[2177] The server analyzes the issues and solutions based on the workflow diagram. The data analysis module analyzes the workflow diagram and identifies issues at specific steps. For example, an issue such as "data checking takes a long time" is extracted. This issue and solution become the input data for the next generated slide.

[2178] Step 17:

[2179] The terminal presents the problem and the solution to the user. The problem and the solution received from the server are displayed on the screen so that the user can confirm them. This allows the user to decide on the next action.

[2180] Step 18:

[2181] The user checks the problem and solution and gives instructions for corrections as necessary. For example, the user may give verbal instructions such as, "It would be better to use the current tool, so please think of another solution." This additional instruction becomes prompt data for the next step.

[2182] Step 19:

[2183] The server generates presentation slides based on the business flow diagram, issues, and solutions. The slide generation module embeds the business flow diagram, issues, and solutions into a template and generates a slide-format file. The output is a PowerPoint or PDF.

[2184] Step 20:

[2185] The server generates a video based on the slides and sends it to the device. The video generation module animates the generated slides, synthesizes the narration from the text, and creates a video file. This video becomes the final output data for the user.

[2186] Step 21:

[2187] The user reviews the final slides and video, and downloads or shares them as needed. The device displays the slides and video, allowing the user to review them and proceed to the next step in the process.

[2188] (Application example 1)

[2189] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2190] Optimizing business processes in factories is important from the perspective of improving efficiency and productivity on-site, but conventional methods require specialized knowledge and require a great deal of time and effort. In particular, it has been difficult to grasp the real-time progress of work within a factory and immediately present optimization proposals. In addition, organizing business flows and creating presentation materials also require time and effort, which leads to a decline in work efficiency. For this reason, there is a need for a system that can efficiently manage business processes within a factory and make immediate proposals.

[2191] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2192] In this invention, the server includes: means for converting a work flow verbally explained by a user into text data using voice recognition technology; means for analyzing the text data to extract workflow steps and generate a workflow diagram; means for converting verbal corrections and supplementary information added by the user into text data using voice recognition technology again to update the workflow diagram; means for automatically analyzing and presenting problems and solutions based on the generated workflow diagram; means for automatically generating presentation images based on the workflow diagram, problems, and solutions; means for creating videos based on the automatically generated images; means for using a robot equipped with a voice recognition module that allows a user to verbally explain factory operations and work instructions in real time; and means for analyzing voice data recorded by the robot and providing optimization suggestions based on the workflow diagram. This enables efficient management of factory business processes and real-time optimization suggestions on-site.

[2193] "User" refers to a person who uses the system to verbally explain the workflow and provide corrections and supplementary information.

[2194] "Verbal explanation" refers to the act of using voice to communicate the flow of work and correction information.

[2195] "Voice recognition technology" refers to the technology that analyzes voice and converts it into text data.

[2196] "Text data" refers to text data converted from speech using speech recognition technology.

[2197] "Workflow" refers to a series of steps or procedures for accomplishing a particular task or work.

[2198] A "step" refers to an individual unit of work within a workflow.

[2199] A "workflow diagram" refers to a visual representation of a workflow.

[2200] A "problem" refers to a glitch or inefficiency that occurs within a workflow.

[2201] A "solution" refers to a specific method or proposal for solving a problem.

[2202] "Presentation images" refers to diagrams and presentation slides that visually illustrate workflows, business problems, and solutions.

[2203] "Video creation" refers to the process of creating visual explanations in video format based on presentation images.

[2204] A "robot equipped with a voice recognition module" refers to a robot that has the ability to record voice data and analyze the voice in real time.

[2205] "Optimization proposals based on workflow diagrams" refers to proposals that analyze workflow diagrams to identify problems and show how to improve them.

[2206] "Real-time" refers to processing and providing information immediately, without delay.

[2207] This invention is a system for efficiently managing work processes in factories and making optimization proposals in real time. This system uses voice reco...

Claims

1. A means for converting the work flow verbally explained by the user into text data using voice recognition technology; A means for analyzing text data to extract steps of a business flow and generate a business flow diagram; A means for converting verbal corrections and supplementary information added by the user into text data using voice recognition technology again and updating the workflow diagram; A means for automatically analyzing and presenting issues and their solutions based on the generated workflow diagram; A means to automatically generate presentation slides based on business flow diagrams, issues, and solutions; A way to create videos based on automatically generated slides, A system including:

2. 2. The system according to claim 1, further comprising means for displaying the generated business flow diagram, the problems, and the solutions on a terminal so that the user can confirm them.

3. The system according to claim 1, further comprising means for creating the generated business flow diagram using a BPMN tool.

Images