System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting text, audio, image, and video data; means for pre-processing and converting the collected data into a standard format; means for analyzing the pre-processed data and suggesting business process optimizations; means for automated document management; and means for customer response automation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Modern companies are increasingly focused on improving business efficiency and reducing costs. However, it is difficult to effectively manage and analyze diverse data, including text, voice, images, and video, and to automate and optimize business processes. Providing fast, personalized customer service is also a major challenge. These complex challenges call for an integrated solution. [Means for solving the problem]

[0005] The present invention provides a system that collects text, audio, image, and video data, preprocesses the collected data and converts it into a standard format, analyzes the preprocessed data, and proposes business process optimization. It also configures a system that includes automated document management and automated customer support. Furthermore, by providing a system that analyzes data using multimodal AI and a system that performs semantic analysis of text data using natural language processing, it achieves business efficiency and high-quality customer service.

[0006] "Text data" refers to data recorded in the form of a string of characters, and may take the form of a document, email, cloud memo, or the like.

[0007] "Voice data" refers to digital data that records human voices and other sounds, such as those used to record telephone conversations with customers and feedback.

[0008] "Image data" refers to digital data that contains visual information, such as photographs or scanned documents, including product images and logos.

[0009] "Video data" refers to digital data recorded in a video format, including presentations, advertising videos, and the like.

[0010] "Collection means" refers to methods and devices for collecting text, audio, image, and video data from various data sources.

[0011] "Preprocessing procedures" refer to a series of data cleaning and standardization operations performed to convert collected data into a format that is easier to analyze.

[0012] "Analytical tools" refer to methods and algorithms for using preprocessed data to extract useful information and insights.

[0013] "Business process optimization measures" refer to proposals and implementation measures to improve business flow and increase efficiency based on analysis results.

[0014] "Document management tools" refer to methods and systems for organizing, classifying, and storing large amounts of document data, and for quickly retrieving and using the information you need.

[0015] "Customer response tools" refer to technologies and methods for generating automated responses to customer inquiries and providing personalized responses.

[0016] "Multimodal AI" is an artificial intelligence technology that comprehensively analyzes multiple data modalities such as text, audio, images, and video.

[0017] "Natural language processing" refers to artificial intelligence techniques for analyzing, understanding, and generating human language. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] This invention is an AI-driven automation system aimed at improving business efficiency in companies. In the embodiment of the invention, a method for integrating and analyzing text, audio, image, and video data to automate business process optimization, document management, and customer support is described.

[0040] System Configuration

[0041] The system mainly consists of three components: a server, a terminal, and a user. The server is primarily responsible for data collection, preprocessing, analysis, optimization, management, and customer support. The terminal is a device that inputs and outputs data and provides a user interface. The user operates the system, uploads data, and reviews and approves proposal results.

[0042] Program processing overview

[0043] Data collection

[0044] The server periodically collects text, audio, image, and video data from various data sources, such as email servers, CRM systems, and customer support recording databases. Users use devices to upload data such as customer feedback and internal reports.

[0045] Data Preprocessing

[0046] The server preprocesses the collected data. For example, it transcribes audio data and standardizes the resolution of image data. It normalizes text data and removes unnecessary characters and whitespace. This ensures the data is in a consistent format and suitable for analysis.

[0047] Data analysis

[0048] The preprocessed data is then analyzed by the server. Natural language processing is used to perform semantic analysis of the text data, and sentiment analysis and topic models are built. Image and video data are also analyzed using computer vision technology to extract the necessary information. This allows issues within the business process to be identified.

[0049] Business process optimization

[0050] Based on the analysis results, the server generates business process optimization proposals. For example, if many customers report the same problem, it will identify the cause and propose improvements. Users can review the proposals on their devices and approve or modify them as necessary.

[0051] Document Management

[0052] The server uses an automatic document management system to efficiently manage internal and external documents. Uploaded documents are automatically categorized and placed in relevant folders. Users can easily access the documents they need using tag or keyword searches.

[0053] Automating customer interactions

[0054] The terminal receives inquiries from customers and immediately sends them to the server, which uses natural language understanding technology to analyze the inquiries and generate appropriate responses. Responses are automatically sent, and users can review and modify them as needed.

[0055] Specific examples

[0056] For example, if a company receives similar complaints from multiple customers at its support desk, the system works as follows:

[0057] 1. The server periodically collects new complaint emails from the support mailbox.

[0058] 2. The server preprocesses the email data and normalizes the text data.

[0059] 3. The server uses natural language processing to analyze the content of the complaint and extract key topics and frequently occurring keywords.

[0060] 4. Based on the analysis results, the server identifies where problems are concentrated in specific products or services and generates improvement proposals.

[0061] 5. The user checks the proposed improvements on the device and approves them if necessary.

[0062] 6. Complaint data will be automatically categorized and saved in relevant folders for future reference.

[0063] 7. When a new inquiry comes in from a customer, the server generates an automatic response and replies to the customer quickly.

[0064] By implementing this invention, companies can significantly improve their business efficiency and increase customer satisfaction.

[0065] The processing flow will be explained below.

[0066] Step 1:

[0067] The server collects data from various sources: email data from the mail server, customer interaction data downloaded from the CRM system, and audio data from the recording database.

[0068] Step 2:

[0069] Users upload data to the system using their devices. Specifically, they drag and drop customer feedback audio files and image data onto a dedicated upload interface to send them to the server.

[0070] Step 3:

[0071] The server preprocesses the collected data in bulk. For example, it uses a speech recognition API to convert voice data into text, an image conversion library to standardize image resolution, and normalizes the text data to remove unnecessary characters and spaces.

[0072] Step 4:

[0073] The server analyzes the preprocessed data. For text data, natural language processing (NLP) is used to perform sentiment analysis and build topic models. Specifically, Python NLP libraries (e.g., NLTK, spaCy) are used to analyze the meaning of the text. For image and video data, computer vision techniques are used, such as OpenCV and TensorFlow, to extract features for facial recognition and object detection.

[0074] Step 5:

[0075] The server generates business process optimization proposals based on the analysis results. For example, if there are many complaints about a particular product, it uses a rule engine to identify the cause and propose improvement measures.

[0076] Step 6:

[0077] The user checks the business process optimization proposals from the server through their terminal. Specifically, an interface is provided for reviewing the proposals and approving or adjusting them as necessary. If the user is satisfied with the proposals, they click the approval button to put them into action.

[0078] Step 7:

[0079] The server automatically classifies and manages documents. For example, it automatically analyzes uploaded documents such as contracts and customer feedback documents, attaches relevant tags, and moves them to the appropriate folders, allowing users to search more efficiently.

[0080] Step 8:

[0081] Users can search for and access the documents they need. Specifically, they can use keyword search and tag filter functions to easily find the documents they are looking for. They can then select the documents they need from the search results, access them, and obtain the information they need.

[0082] Step 9:

[0083] The terminal receives inquiries from customers, analyzes the content using natural language understanding (NLU) technology, and sends the inquiry content to the server.

[0084] Step 10:

[0085] The server generates an appropriate response based on the inquiry. Specifically, it searches for similar cases using past inquiry data and a knowledge base, and generates an automatic response. The generated response is automatically sent to the customer.

[0086] Step 11:

[0087] The user checks the automated response generated by the server, makes any necessary corrections, and then sends the final reply. Specifically, the user checks the response text on the terminal interface, corrects it, and then clicks the send button to provide accurate information to the customer.

[0088] By executing each processing step sequentially in this way, companies can achieve efficient data management and analysis, optimize automated business processes, and respond quickly and appropriately to customers.

[0089] Example 1

[0090] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0091] Modern companies are required to efficiently collect and analyze information from diverse data sources and quickly and accurately optimize their business processes. However, conventional systems require a great deal of time and effort to integrate and analyze data in different formats, and there are a lack of efficient ways to improve business processes. Furthermore, automation of document management and customer support is insufficient, resulting in increased effort. The purpose of this invention is to solve these problems.

[0092] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0093] In this invention, the server includes means for collecting text, audio, image, and video data from various data sources, means for preprocessing the collected data, transcribing it, standardizing its resolution, normalizing it, and converting it into a standard format, means for analyzing the preprocessed data and proposing business process optimization using natural language processing and computer vision technologies, means for performing automated document management, converting documents into text using OCR technology, and classifying them, and means for automating customer support using natural language understanding technology. This makes it possible to efficiently integrate and analyze data in different formats, quickly optimize business processes, and automate document management and customer support.

[0094] A "data source" is a source of information used by a system to gather information.

[0095] "Preprocessing" refers to a series of processes to format collected data so that it is easier to analyze.

[0096] "Transcription" is the process of converting audio data into text data.

[0097] "Resolution unification" is a process of unifying the resolution of image data to a certain standard.

[0098] "Normalization" is the process of removing unnecessary characters and spaces from text data and formatting it into a consistent format.

[0099] "Natural language processing" is a technology that allows computers to understand and analyze human language and extract meaning.

[0100] "Computer vision technology" is a technology that analyzes image and video data to extract necessary information.

[0101] "Business process optimization" is the process of proposing improvements and streamlining of business processes based on the results of analysis.

[0102] "Automated document management" is a system that automates processes such as document classification, storage, and retrieval.

[0103] "OCR technology" is an abbreviation for Optical Character Recognition, and is a technology that extracts character information from image data.

[0104] "Natural language understanding technology" is a technology that allows computers to understand human language and generate appropriate responses and actions.

[0105] "Customer response automation" is the process by which a system automatically generates and responds to customer inquiries.

[0106] This invention is an AI-driven automation system that collects text, audio, image, and video data from multiple data sources, and preprocesses and analyzes this data to optimize corporate business processes.

[0107] System Configuration

[0108] This system is mainly composed of three components: a server, a terminal, and a user. The server is primarily responsible for data collection, preprocessing, analysis, business process optimization, document management, and customer support. The terminal functions as the user interface, inputting and outputting data. The user operates the system, uploading data, and confirming and approving proposal results.

[0109] System action

[0110] Data collection

[0111] The server periodically collects text, audio, image, and video data from data sources such as email servers, CRM systems, and customer support recording databases. The specific software used includes data collection agents and API interfaces. Users use devices to upload data, such as customer feedback and internal reports, to the system.

[0112] Data Preprocessing

[0113] The server preprocesses the collected data. For audio data, it transcribes it using speech recognition software (e.g., speech recognition API). For image data, it uses image processing software (e.g., OpenCV) to standardize the resolution. For text data, it normalizes it using a natural language processing toolkit (e.g., NLTK) to remove unnecessary characters and spaces.

[0114] Data analysis

[0115] The preprocessed data is then analyzed by the server. Natural language processing tools (e.g., spaCy) are used to perform semantic analysis on text data, and sentiment analysis and topic models are built. Image and video data are analyzed using computer vision techniques (e.g., TensorFlow) to extract the necessary information. This allows issues within the business process to be identified.

[0116] Business process optimization

[0117] Based on the analysis results, the server generates business process optimization proposals. If many customers have reported similar problems, it will identify the cause and propose specific improvement measures (for example, updating FAQs or changing processes). Users can check the proposals on their devices and approve or modify them as necessary.

[0118] Document Management

[0119] The server uses an automated document management system (e.g., a document management platform) to efficiently manage internal and external documents. Uploaded documents are converted into text using OCR technology (e.g., optical character recognition tools), and automatically classified and stored. Users can easily access the documents they need using tag and keyword searches.

[0120] Automating customer interactions

[0121] The terminal receives inquiries from customers and immediately sends the details to the server. The server uses natural language understanding technology (e.g., language understanding models) to analyze the inquiry and automatically generate an appropriate response. Responses are sent automatically, but users can review and modify complex inquiries.

[0122] Specific examples

[0123] For example, if a company receives similar complaints from multiple customers at its support desk, the system works as follows:

[0124] 1. The server periodically collects new complaint emails from the support mailbox.

[0125] 2. The server preprocesses the email data and normalizes the text data.

[0126] 3. The server uses natural language processing to analyze the content of the complaint and extract key topics and frequently occurring keywords.

[0127] 4. Based on the analysis results, the server identifies where problems are concentrated in specific products or services and generates improvement proposals.

[0128] 5. The user checks the proposed improvements on the device and approves them if necessary.

[0129] 6. Complaint data will be automatically categorized and saved in relevant folders for future reference.

[0130] 7. When a new inquiry comes in from a customer, the server generates an automatic response and replies to the customer quickly.

[0131] Prompt Sentence Examples

[0132] Example of an input prompt for a generative AI model:

[0133] "Generate optimization proposals from the following workflow:

[0134] 1. Receiving and processing customer complaints

[0135] 2. Internal Quality Assurance Process

[0136] 3. Product Development Cycle

[0137] Please provide a summary of your analysis and specific suggestions for improvement.

[0138] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0139] Step 1: Data collection

[0140] The server periodically collects text, voice, image, and video data from data sources such as email servers, CRM systems, and customer support recording databases. The server retrieves this data through an API interface or by using a data collection agent and stores it in data storage. Users use their devices to upload data such as customer feedback and internal reports and send it to the server. The input of this step is raw data such as emails, voice recordings, and images, and the output is the collected, unprocessed data.

[0141] Step 2: Data Preprocessing

[0142] The server preprocesses the collected raw data. For audio data, it converts it into text using speech recognition software (e.g., a speech recognition API). For image data, it uses image processing software (e.g., OpenCV) to standardize the resolution and perform noise reduction if necessary. Furthermore, it normalizes the text data using a natural language processing toolkit (e.g., NLTK) to remove unnecessary characters and spaces. The input of this step is the raw data, and the output is the preprocessed data.

[0143] Step 3: Data analysis

[0144] The server analyzes the preprocessed data. Text data is semantically analyzed using natural language processing tools (e.g., spaCy) to perform sentiment analysis and build topic models. Image data and video data are analyzed using computer vision techniques (e.g., TensorFlow) to extract the necessary information. The analysis results include the data's topics, sentiment scores, and the detection of specific events and objects. The input to this step is the preprocessed data, and the output is the analysis results.

[0145] Step 4: Optimize business processes

[0146] The server generates optimization proposals for business processes based on the results of data analysis. For example, if many complaints with low sentiment scores are concentrated around a specific product, the server identifies the cause and proposes countermeasures (e.g., FAQ updates, product modifications). The user can review these proposals on their device and approve or modify them as necessary. The input to this step is the analysis results, and the output is optimization proposals.

[0147] Step 5: Document Management

[0148] The server uses an automated document management system (e.g., a document management platform) to efficiently manage internal and external documents. It converts uploaded documents into text using OCR technology (e.g., optical character recognition tools), classifies them based on categories, and stores them in relevant folders. Users can access the documents they need using a search interface. The input of this step is document data, and the output is classified and stored documents.

[0149] Step 6: Automate customer interactions

[0150] The terminal receives customer inquiries in real time and immediately sends the content to the server. The server uses natural language understanding technology (e.g., a language understanding model) to analyze the inquiry and generate an appropriate automated response. For simple inquiries, an automated response is sent to the customer immediately, but for complex inquiries, the user can review and modify the response. The input to this step is the text of the customer inquiry, and the output is the generated response.

[0151] (Application example 1)

[0152] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0153] In manufacturing processes within factories, the collection and analysis of a wide variety of data (text, audio, images, video) is not being carried out efficiently, making it difficult to optimize business processes and not fully automating document management and customer support. This makes it difficult to improve business efficiency and customer satisfaction. To solve this problem, an automated system that uses integrated data management and advanced analysis technology is required.

[0154] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0155] In this invention, the server includes a means for collecting text, audio, image, and video data, a means for preprocessing the collected data and converting it into a standard format, and a means for analyzing the preprocessed data and proposing business process optimization. This enables data analysis and optimization proposals to be made using a clustering algorithm in a manufacturing environment. The server also includes a means for converting audio data into text data and analyzing it, and a means for collecting and preprocessing data from various sensors, cameras, and audio recorders. Furthermore, by incorporating a means for analyzing the sentiment of audio data using natural language processing technology and improving the efficiency of customer service, it is possible to improve not only business efficiency but also customer satisfaction.

[0156] "Text data" refers to character string information written in a natural language, and includes formats such as documents, emails, and reports.

[0157] "Voice data" refers to digitally recorded human speech or acoustic signals, and is used for speech recognition and emotion analysis.

[0158] "Image data" is visual information stored in digital form, including photographs, graphics, and diagrams.

[0159] "Moving image data" refers to data that digitally records visual information that changes over time, including video clips and video recordings.

[0160] "Preprocessing" refers to a series of operations to convert collected data into a format suitable for analysis, including format standardization and noise removal.

[0161] A "clustering algorithm" is a machine learning technique for classifying data into multiple clusters, based on patterns and similarities in the data.

[0162] "Natural language processing" is a technology that enables computers to understand and analyze human language, and is used for semantic and sentiment analysis of text.

[0163] "Sentiment analysis" is a technology that detects and classifies emotions from text and voice data, and is useful for automating customer responses.

[0164] A "sensor" is a device that detects the physical state of an environment or object and is used to collect data.

[0165] A "camera" is a device that captures visual information and stores it as digital data, and is used to collect image data and video data.

[0166] "Audio recording device" means a device for recording audio in digital form and is used to collect speech and acoustic data.

[0167] This invention is an AI-driven automation system that aims to improve the efficiency of work processes in a factory. In an embodiment of the invention, the system is composed of three entities: a server, a terminal, and a user.

[0168] System Configuration

[0169] The server plays the primary role of collecting data, preprocessing, analyzing, optimizing, managing, and responding to customers. The terminal is a device that inputs and outputs data and provides a user interface, and is also used for robots installed on production lines in factories. The user operates the system, uploads data, and checks and approves proposed results.

[0170] Program processing overview

[0171] Data collection

[0172] The server periodically collects text data, audio data, image data, and video data from various sensors, cameras, and audio recorders in the factory. For example, the server acquires this data from the production line's recording devices and quality control systems. Users use their terminals to upload data such as production daily reports and error logs.

[0173] Data Preprocessing

[0174] The server preprocesses the collected data. For example, it converts audio data to text data and standardizes the resolution of image data. Text data is normalized and unnecessary characters and spaces are removed. This ensures that the data is in a consistent format and suitable for analysis.

[0175] Data analysis

[0176] The preprocessed data is then analyzed by the server. Natural language processing is used to perform semantic and sentiment analysis of the text data and build a topic model. A clustering algorithm is also used to classify the manufacturing data into clusters. Image and video data is analyzed using computer vision technology to extract the necessary information. This allows issues in the manufacturing process within the factory to be identified.

[0177] Business process optimization

[0178] Based on the analysis results, the server generates business process optimization proposals. For example, if multiple manufacturing errors occur for the same reason, the server will identify the cause and propose improvement measures. Users can check the proposals on their devices and approve or modify them as necessary.

[0179] Document Management

[0180] The server uses an automatic document management system to efficiently manage documents inside and outside the factory. Uploaded documents are automatically categorized and placed in relevant folders. Users can easily access the documents they need using tag or keyword searches.

[0181] Automating customer interactions

[0182] The terminal receives inquiries from customers and immediately sends them to the server, which uses natural language understanding technology to analyze the inquiries and generate appropriate responses. Responses are automatically sent, and users can review and modify them as needed.

[0183] Specific examples

[0184] As a concrete example, when quality control is performed on a manufacturing line in a factory, the system operates as follows.

[0185] 1. The server periodically collects new data from the production line's sensor data and quality control records.

[0186] 2. The server preprocesses these data and converts them into a standard format.

[0187] 3. The server analyzes the data using natural language processing and clustering algorithms to identify key manufacturing challenges and frequent problems.

[0188] 4. Based on the analysis results, the server identifies where problems are concentrated in specific manufacturing processes and generates improvement proposals.

[0189] 5. The user checks the proposed improvements on the device and approves them if necessary.

[0190] 6. Manufacturing data is automatically categorized and saved in relevant folders for future reference.

[0191] Prompt Sentence Examples

[0192] "Collect new complaint emails from the support mailbox, analyze the content of the complaint using natural language processing, and generate a report of the results."

[0193] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0194] Step 1:

[0195] Data collection

[0196] The server periodically collects text data, voice data, image data, and video data from various sensors, cameras, and audio recorders in the factory. For example, the server receives inputs from the production line monitoring system and quality control system to acquire these data, and outputs the collected data to be stored in the centralized management system.

[0197] Step 2:

[0198] Data Preprocessing

[0199] The server preprocesses the collected data. For example, it converts audio data into text data and standardizes the resolution of image data. Specifically, it uses voice recognition software to convert audio data into text and image processing algorithms to standardize the resolution of images. This preprocessing ensures that the data is in a consistent format, providing appropriate input data for analysis. The output is standardized text, audio, image, and video data.

[0200] Step 3:

[0201] Data analysis

[0202] The preprocessed data is then analyzed by the server, which uses natural language processing to perform semantic and sentiment analysis of the text data. For example, a generative AI model is used to detect sentiment from the text data and build a topic model. A clustering algorithm is also used to classify the manufacturing data into clusters. Image and video data is analyzed using computer vision techniques to extract information for anomaly detection and quality assessment on the production line. The output of this step is a detailed report based on the analysis results.

[0203] Step 4:

[0204] Business process optimization

[0205] Based on the analysis results, the server generates optimization proposals for business processes. The server identifies patterns of manufacturing errors and frequent problems and proposes specific improvement measures. It also uses a customized generative AI model to generate optimization proposals. These optimization proposals include specific instructions and action plans for improving the manufacturing process, thereby enabling efficient business operations. The output is a proposal for improving the business process.

[0206] Step 5:

[0207] Document Management

[0208] The server uses an automatic document management system to efficiently manage documents inside and outside the factory. Uploaded documents are automatically categorized and placed in relevant folders. They are also designed to be easily accessible using tag and keyword searches, allowing users to quickly access the documents they need. This improves the efficiency of document management. The output is categorized and organized document data.

[0209] Step 6:

[0210] Automating customer interactions

[0211] The terminal receives an inquiry from a customer and immediately sends the content to the server. The server uses natural language understanding technology to analyze the content of the inquiry and generate an appropriate response. The generated response is sent automatically, but the user can confirm and modify it if necessary. A prompt sentence (e.g., "Please collect new complaint emails from the support mailbox, analyze the content of the complaint using natural language processing, and generate a report of the results") may also be used to generate an initial response. The output of this step is a prompt and appropriate response to improve customer satisfaction.

[0212] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0213] This invention is an AI-driven automation system aimed at improving the efficiency of corporate operations, and in particular, by combining it with an emotion engine that recognizes user emotions, it achieves more advanced optimization of business processes and automation of customer support. In the embodiments of the present invention, a method is described for integrating and analyzing text, audio, image, and video data to automate optimization of business processes, document management, and customer support that take user emotions into consideration.

[0214] System Configuration

[0215] The system mainly consists of three components: a server, a terminal, and a user. The server is primarily responsible for data collection, preprocessing, analysis, optimization, management, customer service, and emotion recognition. The terminal is a device that inputs and outputs data and provides a user interface. The user operates the system, uploads data, and reviews and approves proposal results.

[0216] Program processing overview

[0217] Data collection

[0218] The server collects data from various sources, including email servers, CRM systems, and customer support recording databases, periodically retrieving text, audio, image, and video data. Users use their devices to upload data, such as customer feedback and internal reports, to the system.

[0219] Data Preprocessing

[0220] The server preprocesses the collected data and converts it into a standard format. For example, audio data is transcribed, image data is standardized, and text data is normalized to remove unnecessary characters and whitespace. This ensures the data is in a consistent format and suitable for analysis.

[0221] Data Analysis and Emotion Recognition

[0222] The preprocessed data is then analyzed by the server. Natural language processing is used to perform semantic analysis of the text data, and sentiment analysis is also performed simultaneously. For example, positive, negative, and neutral emotions are extracted from the text data. Voice data is also analyzed using a sentiment extraction engine to recognize the customer's emotions. Image and video data are also analyzed using computer vision technology to identify emotions from the user's facial expressions.

[0223] Business process optimization

[0224] Based on the analysis results and the recognized emotions, the server generates business process optimization proposals. For example, if many customers have negative emotions about a particular product, the server will identify the cause and propose improvements. The user can review the proposals on their device and approve or modify them as necessary.

[0225] Document Management

[0226] The server uses an automatic document management system to efficiently manage internal and external documents. Uploaded documents are automatically categorized and placed in relevant folders. Users can easily access the documents they need using tag or keyword searches.

[0227] Automating customer interactions

[0228] The terminal receives inquiries from customers and immediately sends them to the server. The server uses natural language understanding technology to analyze the inquiries and generate appropriate responses. Responses can also be tailored based on the user's sentiment. Responses are automatically sent, and users can review and modify them as needed.

[0229] Specific examples

[0230] For example, if a company receives similar complaints from multiple customers at its support desk, the system works as follows:

[0231] 1. The server periodically collects new complaint emails from the support mailbox.

[0232] 2. The server preprocesses the email data and normalizes the text data.

[0233] 3. The server uses natural language processing to analyze the content of the complaint and extract key topics and frequently occurring keywords.

[0234] 4. At the same time, the server uses an emotion engine to recognize customer emotions from the text data. For example, if many emails express negative emotions, this is detected.

[0235] 5. Based on the analysis results and sentiment data, the server identifies problems concentrated in specific products or services and generates improvement proposals. For example, if there is a lot of negative feedback stating that "there are problems with the quality of a specific product," it will propose improvement measures for that product.

[0236] 6. The user checks the proposed improvements on the terminal and approves them if necessary.

[0237] 7. Complaint data will be automatically categorized and saved in relevant folders for future reference.

[0238] 8. When a new inquiry comes in from a customer, the server generates an automatic response and quickly replies to the customer with a message that reflects the user's feelings.

[0239] By implementing this invention, enterprises can significantly improve their business efficiency and provide high-quality customer service that takes into consideration users' emotions.

[0240] The processing flow will be explained below.

[0241] Step 1:

[0242] The server collects data from various sources: email data from the mail server, customer interaction data from the CRM system, and voice data from the recording database. The server periodically checks these data sources and collects any new data.

[0243] Step 2:

[0244] Users upload data to the system using their devices. Specifically, they drag and drop customer feedback audio files and image data into a dedicated upload interface and send them to the server. Users do this as part of their daily work.

[0245] Step 3:

[0246] The server preprocesses the collected data. For example, it transcribes audio data and standardizes the resolution of image data. It normalizes text data and removes unnecessary characters and whitespace. This preprocessing ensures that the data is in a consistent format and suitable for analysis.

[0247] Step 4:

[0248] The server analyzes the preprocessed data. Specifically, it uses natural language processing (NLP) to perform semantic analysis of the text data. It uses Python NLP libraries (e.g., spaCy, NLTK) to analyze the structure of the text and build sentiment analysis and topic models. It also uses speech recognition and emotion recognition engines to extract emotions from audio data.

[0249] Step 5:

[0250] The server analyzes image and video data. It uses computer vision technology (e.g., OpenCV, TensorFlow) to analyze the image data and identify emotions from the user's facial expressions. Similarly, it extracts emotions from video data based on specific facial expressions and movements.

[0251] Step 6:

[0252] The server uses an emotion recognition engine to extract the user's emotion from the analyzed data. For text data, it calculates an emotion score such as positive, negative, or neutral, and generates similar emotion scores for audio and image data.

[0253] Step 7:

[0254] The server generates business process optimization proposals based on the analysis results and sentiment data. For example, if many customers have negative feelings about a particular product, the server identifies the cause and proposes improvement measures. The proposals are generated in the form of specific action plans and changes.

[0255] Step 8:

[0256] The user checks the business process optimization proposals from the server through their terminal. For example, they review the proposals displayed on the terminal and approve or adjust them as necessary. If the user is satisfied with the proposals, they click the approval button to put them into action.

[0257] Step 9:

[0258] The server automatically classifies and manages documents. For example, it automatically analyzes uploaded documents such as contracts and customer feedback documents, attaches relevant tags, and moves them to the appropriate folders, improving document search efficiency.

[0259] Step 10:

[0260] Users search for and access the documents they need. Specifically, they use the device's search function to enter keywords or tags to quickly identify the desired document. They then select the relevant document from the search results and access it to obtain the information they need.

[0261] Step 11:

[0262] The terminal receives inquiries from customers, analyzes the content using natural language understanding (NLU) technology, and sends the inquiry to a server. For example, this is done by chatbots and email reception interfaces.

[0263] Step 12:

[0264] The server generates an appropriate response based on the inquiry. Specifically, it searches for similar cases using past inquiry data and a knowledge base, and generates an automatic response. The generated response is quickly sent to the customer.

[0265] Step 13:

[0266] The user checks the automated response generated by the server, makes any necessary corrections, and then sends the final reply. For example, by checking the response text on the terminal interface, correcting it, and then clicking the send button, the user can provide accurate information to the customer.

[0267] Example 2

[0268] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0269] Modern business activities require the efficient management and analysis of massive amounts of data, as well as the optimization of business processes and the automation of customer responses. In particular, an approach that handles text, voice, image, and video data in an integrated manner and takes user emotions into account is essential, but there are currently no systems that can effectively achieve this. Furthermore, managing and analyzing data manually is time-consuming, costly, and prone to human error. Therefore, there is a need for a comprehensive system that automates everything from data collection and analysis to optimization proposals, document management, and customer responses.

[0270] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0271] In this invention, the server includes means for collecting text, audio, image, and video data from various data sources, means for preprocessing the collected data and converting it into a standard format, and means for analyzing the preprocessed data using a natural language processing and emotion recognition engine. This enables companies to efficiently manage and analyze massive amounts of data, optimize business processes, and automate customer responses.

[0272] "Various data sources" refers to multiple information sources, including email servers, customer relationship management systems, customer support recording databases, etc.

[0273] "Text data" refers to data expressed in text format, such as emails, reports, and chat history.

[0274] "Audio data" refers to data expressed in the form of voice, such as audio files and recorded data.

[0275] "Image data" refers to data expressed in a visual format, such as photographs, screenshots, and illustrations.

[0276] "Video data" refers to data that is presented in a moving visual format, such as a video clip or recorded data.

[0277] "Collect" refers to the act of obtaining data and putting it into a system.

[0278] "Preprocessing" refers to the initial processing of collected data to convert it into a form that is easier to analyze. Examples include transcribing audio data and normalizing text data.

[0279] "Standard format" refers to a standard for standardizing data into a consistent format.

[0280] "Natural language processing" refers to the technology that enables computers to understand and analyze natural language written and spoken by humans.

[0281] An "emotion recognition engine" refers to software or algorithms that identify and analyze emotions contained within data.

[0282] "Analyzing" refers to the act of analyzing data to understand its content and patterns and extract information.

[0283] "Business process optimization" refers to the act of planning and implementing proposals and measures to improve the efficiency of corporate activities.

[0284] "Automated document management" refers to the automatic performance of management tasks such as document classification, storage, and retrieval by a computer system.

[0285] "Customer response automation" refers to a system automatically generating and sending appropriate responses to customer inquiries.

[0286] This invention is an AI-driven automation system aimed at improving business efficiency, and in particular, by combining an emotion engine that recognizes user emotions, it optimizes business processes and automates customer support. This system is mainly composed of three components: a server, a terminal, and a user. Specific embodiments of this system are described below.

[0287] System Configuration

[0288] The system consists of three components: a server, a terminal, and a user. The server is primarily responsible for data collection, preprocessing, analysis, optimization, management, customer support, and emotion recognition. The terminal is a device that inputs and outputs data and provides a user interface. The user operates the system, uploads data, and confirms and approves proposal results. Specific implementation examples of each component are shown below.

[0289] server

[0290] The server periodically collects text, audio, image, and video data from various data sources (e.g., email servers, CRM systems, customer support recording databases, etc.). The collected data is ingested through an ETL (Extract, Transform, Load) process. The server then preprocesses the collected data. Audio data is transcribed using a speech-to-text engine (e.g., Google Cloud Speech-to-Text), image data is standardized in resolution, and video data is broken down into frames. Text data is normalized, and unnecessary characters and whitespace are removed. Regular expressions and natural language processing tools (e.g., NLTK) are used in this process.

[0291] The preprocessed data is then analyzed using a natural language processing (NLP) engine (e.g., BERT or GPT model) to perform semantic analysis of the text data. A sentiment analysis engine (e.g., Sentiment140) is used to extract positive, negative, or neutral sentiment. Audio data is also analyzed using a sentiment extraction engine. Finally, image and video data are analyzed using computer vision techniques (e.g., OpenCV or deep learning models) to identify emotions from the user's facial expressions.

[0292] Based on the analysis results, the server generates business process optimization proposals. For example, if many customers have negative feelings about a particular product, it will suggest improvements for that product. It also uses an automated document management system to efficiently manage uploaded documents. Documents are categorized into appropriate folders, and necessary documents can be quickly accessed using tag or keyword searches.

[0293] Terminal

[0294] The terminal is a device that allows users to input and output data and provides a user interface. Users upload data through the terminal and review and approve proposals and analysis results. When users upload data, the terminal checks the data format and displays a warning if the format is inappropriate. The terminal also receives inquiries from customers and immediately sends the details to the server.

[0295] User

[0296] Users operate the system to upload data and review and approve proposal results. Specifically, they use terminals to upload data such as customer feedback and internal reports to the system. They also review the proposal content and approve or modify it if necessary. When a customer inquiry is received, they can also review the generated response and make modifications as necessary.

[0297] Specific examples

[0298] For example, if a company receives similar complaints from multiple customers at its support desk, the system works as follows:

[0299] 1. The server periodically collects new complaint emails from the support mailbox.

[0300] 2. The server preprocesses the email data and normalizes the text data.

[0301] 3. The server uses natural language processing to analyze the content of the complaint and extract key topics and frequently occurring keywords.

[0302] 4. At the same time, the server uses an emotion engine to recognize customer emotions from the text data. For example, if many emails express negative emotions, this is detected.

[0303] 5. Based on the analysis results and sentiment data, the server identifies problems concentrated in specific products or services and generates improvement proposals. For example, if there is a lot of negative feedback stating that "there are problems with the quality of a specific product," it will propose improvement measures for that product.

[0304] 6. The user checks the proposed improvements on the terminal and approves them if necessary.

[0305] 7. Complaint data will be automatically categorized and saved in relevant folders for future reference.

[0306] 8. When a new inquiry comes in from a customer, the server generates an automatic response and quickly replies to the customer with a message that reflects the user's feelings.

[0307] Prompt Sentence Examples

[0308] Below are some examples of text data to analyze and specific prompts to extract sentiment:

[0309] Prompt: "Conduct a customer feedback analysis and generate a report that extracts key topics and their respective sentiments. Analyze the following feedback: 'Product X is difficult to use. Support is slow to respond.'"

[0310] By feeding this prompt into a generative AI model, the system can analyze the main topics of the feedback (e.g., usability, support response) and sentiment (negative) and generate a corresponding report.

[0311] In this way, the present invention significantly improves the business efficiency of companies, and in particular enables high-quality responses that take into account the user's emotions.

[0312] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0313] Step 1:

[0314] The server collects text, audio, image, and video data from various data sources (e.g., email servers, CRM systems, customer support recording databases, etc.) and periodically ingests new data by running an ETL (Extract, Transform, Load) process. This step uses data from various data sources as input and produces the collected data as output.

[0315] Step 2:

[0316] The server preprocesses the collected data. For example, audio data is transcribed using a speech-to-text engine, image data is standardized in resolution, and video data is broken down into frames. Text data is normalized using regular expressions and natural language processing tools. This step uses the collected raw data as input and produces preprocessed data as output.

[0317] Step 3:

[0318] The server analyzes the preprocessed data. It uses a natural language processing engine (e.g., a GPT model) to perform semantic analysis of the text data and a sentiment analysis engine to extract positive, negative, or neutral sentiment. Audio data is also analyzed using a sentiment extraction engine, and image and video data are analyzed using computer vision techniques. This step uses the preprocessed data as input and generates analysis results as output.

[0319] Step 4:

[0320] The server generates business process optimization proposals based on the analysis results. For example, if a particular product has received a lot of negative feedback, it generates improvement proposals for that product. It also includes automated processes for customer support. This step uses the analysis results as input and generates optimization proposals and automated responses as output.

[0321] Step 5:

[0322] The user uses a terminal to review the improvement proposals and the automated response. The terminal displays the proposals and provides an interface for the user to approve or modify them as needed. If the user makes modifications, the modified proposals are fed back into the system. This step uses the optimization proposals and automated response as inputs and generates the proposals that have been reviewed and modified by the user as output.

[0323] Step 6:

[0324] The server uses an automated document management system to classify and store the uploaded documents. The documents are categorized into appropriate folders and can be efficiently accessed using tag and keyword searches. This step uses the uploaded documents as input and produces classified and stored documents as output.

[0325] Step 7:

[0326] When the terminal receives a customer inquiry, it immediately sends the content to the server, which analyzes the inquiry and generates an appropriate response using natural language understanding technology. The response can also be tailored based on the user's sentiment. This step uses the customer inquiry information as input and provides the generated response as output.

[0327] (Application example 2)

[0328] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0329] In conventional corporate business processes, it was difficult to analyze user emotions in real time and provide optimal responses. Furthermore, when it came to displaying advertisements, static ads were often displayed without considering user emotions, making it impossible to deliver advertisements effectively. This resulted in problems such as lower customer satisfaction and reduced business efficiency.

[0330] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0331] In this invention, the server includes means for collecting text, audio, image, and video data, means for preprocessing the collected data and converting it into a standard format, means for analyzing the preprocessed data and proposing business process optimization, means for automated document management, means for automating customer support, means for tracking user gaze and analyzing emotions in real time, and means for displaying optimal advertisements based on the analyzed emotions. This allows business processes and advertisement display to be optimized based on user emotions, enabling highly efficient and high-quality customer support.

[0332] "Text data" refers to data that contains sentences or character information, and includes emails, chat messages, documents, and the like.

[0333] "Voice data" means data obtained by recording a human voice or other sounds, including voice messages, telephone recordings, and customer support audio recordings.

[0334] "Image data" refers to visual information recorded as still images, including photographs, screenshots, illustrations, etc.

[0335] "Movie data" refers to dynamic visual information consisting of a series of image frames, including video clips and video recordings.

[0336] "Collection methods" are the technical means used to obtain text, audio, image, and video data from a variety of sources, including database access and use of APIs.

[0337] "Preprocessing means" refers to technology that converts collected data into a format that is easier to analyze, such as transcribing audio data or standardizing image resolution.

[0338] A "standard format" is a format in which data of different formats is converted into a consistent format that can be analyzed, making data analysis easier.

[0339] "Means of analysis" refers to technologies that use preprocessed data to extract information and optimize or propose business processes, including machine learning and natural language processing.

[0340] "Means for proposing optimization" refers to techniques that propose methods and means for improving the efficiency of business processes based on the analysis results, and includes formulating plans and presenting improvement measures.

[0341] "Means for document management" refers to technology for efficiently managing documents, including automatic classification and tagging of documents, and search functions.

[0342] "Means for automating customer support" refers to technology that automatically responds to inquiries and requests from customers, and includes chatbots and automatic reply systems.

[0343] "Gaze tracking means" refers to technology that detects the direction of a user's gaze and monitors it in real time, and includes gaze tracking cameras and sensors.

[0344] "Means for analyzing emotions in real time" refers to technology that detects the user's emotions at that time from their facial expressions, voice, and text, and includes facial expression recognition engines and emotion analysis algorithms.

[0345] The "means for displaying advertisements" refers to technology that displays appropriate advertisements to users based on the analyzed emotions, and includes displays and marketing engines.

[0346] This invention is an AI-driven automation system aimed at improving business efficiency and optimizing customer service. The system includes means for collecting text, voice, image, and video data, means for preprocessing the collected data and converting it into a standard format, means for analyzing the preprocessed data and proposing business process optimization, means for automated document management, and means for automating customer service. It also includes means for tracking a user's gaze and analyzing their emotions in real time, and means for displaying optimal advertisements based on the analyzed emotions.

[0347] System Configuration

[0348] This system is mainly composed of three components: a server, a terminal, and a user. The server is primarily responsible for data collection, preprocessing, analysis, optimization, emotion recognition, and display of advertisements. The terminal is a device that inputs and outputs data and provides a user interface. The user operates the system, uploads data, and reviews and approves the proposed results.

[0349] Data collection

[0350] The server collects text, audio, image, and video data from various data sources, such as email servers, CRM systems, and customer support recording databases, and users use their devices to upload data to the system, including customer feedback and internal reports.

[0351] Data Preprocessing

[0352] The server preprocesses the collected data and converts it into a standard format. For example, audio data is transcribed, image data is standardized, and text data is normalized to remove unnecessary characters and whitespace. This ensures the data is in a consistent format and suitable for analysis.

[0353] Data Analysis and Emotion Recognition

[0354] The preprocessed data is then analyzed by the server. Natural language processing is used to perform semantic analysis of the text data, and sentiment analysis is also performed simultaneously. For example, positive, negative, and neutral emotions are extracted from the text data. Voice data is also analyzed using a sentiment extraction engine to recognize customer emotions. Image and video data are also analyzed using computer vision technology to identify emotions from the user's facial expressions.

[0355] Business process optimization

[0356] Based on the analysis results and the recognized emotions, the server generates business process optimization proposals. For example, if many customers have negative emotions about a particular product, the server will identify the cause and propose improvements. The user can review the proposals on their device and approve or modify them as necessary.

[0357] Document Management

[0358] The server efficiently manages internal and external documents using an automatic document management system. Uploaded documents are automatically categorized and placed in relevant folders. Users can easily access the documents they need using tag or keyword searches.

[0359] Automating customer interactions

[0360] The terminal receives inquiries from customers and immediately sends them to the server. The server uses natural language understanding technology to analyze the inquiries and generate appropriate responses. It can also adjust the responses based on the user's sentiment. Responses are automatically sent, and users can review and modify them as needed.

[0361] Eye Tracking and Emotion Analysis

[0362] When a user wears smart glasses, the eye tracking camera detects the direction of their gaze and analyzes their facial expressions in real time. The server collects this data and performs emotion analysis. Based on the analyzed emotion, the server selects the most appropriate advertisement and displays it in the user's field of view.

[0363] Specific examples

[0364] When a user wears smart glasses and browses web content, Serve analyzes their gaze and facial expressions in real time. For example, if a user shows interest in a particular product page, Serve can detect positive emotions and display relevant ads based on those emotions. This maximizes the effectiveness of advertising and improves the user experience.

[0365] An example of a prompt is, "Design an application that recognizes emotions in real time from the gaze and facial expressions captured by the camera when the user is wearing smart glasses, and displays the most appropriate advertisement based on that emotion. Specifically, implement a function that uses emotion recognition technology to detect the main emotion and selects and displays an appropriate advertisement image. In addition, the advertisement image will be selected from a list that has been pre-classified into several emotional categories."

[0366] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0367] Step 1: Data collection

[0368] The server periodically retrieves text, audio, image, and video data from mail servers, CRM systems, and customer support recording databases. Users use their devices to upload data, such as customer feedback and internal reports, to the system. The input is various data from external data sources, and the output is the collected raw data. Specifically, the server retrieves data using API calls and database queries.

[0369] Step 2: Preprocessing the data

[0370] The server preprocesses the collected data and converts it into a standard format. Specifically, it transcribes audio data, standardizes the resolution of image data, and normalizes text data to remove unnecessary characters and spaces. The input is the raw data collected, and the output is the preprocessed data. The server performs these conversions using a speech recognition engine and image processing algorithms.

[0371] Step 3: Data analysis and emotion recognition

[0372] The server analyzes the preprocessed data and uses natural language processing to perform semantic analysis of the text data, while simultaneously performing sentiment analysis. For example, positive, negative, and neutral emotions are extracted from the text data. Voice data is analyzed using an emotion extraction engine to recognize customer emotions. Image data and video data are analyzed using computer vision technology. The input is the preprocessed data, and the output is the analysis results and emotional data. Specifically, the server performs these analyses using a natural language processing library and a facial recognition engine.

[0373] Step 4: Optimize business processes

[0374] The server generates optimization proposals for business processes based on the analysis results and the recognized emotions. For example, if many customers have negative emotions about a particular product, the cause can be identified and improvement measures proposed. The input is the analysis results and emotion data, and the output is optimization proposals for business processes. Specifically, the server generates optimization proposals using machine learning algorithms.

[0375] Step 5: Document Management

[0376] The server uses an automated document management system to efficiently manage internal and external documents. Uploaded documents are automatically categorized and placed in relevant folders. Users can access the documents they need using tag or keyword searches. The input is the uploaded document data, and the output is organized and categorized documents. Specifically, the server uses document management software.

[0377] Step 6: Automate customer interactions

[0378] The terminal receives an inquiry from a customer and immediately sends the content to the server. The server uses natural language understanding technology to analyze the inquiry and generate an appropriate response. The input is the customer's inquiry, and the output is the generated response message. Specifically, the server uses a natural language understanding engine.

[0379] Step 7: Gaze Tracking and Emotion Analysis

[0380] When a user wears smart glasses, an eye-tracking camera is used to detect the direction of gaze and analyze facial expressions in real time. The server collects this data and performs emotion analysis. The input is the user's gaze and facial expression data, and the output is analyzed emotion data. Specifically, the server uses an eye-tracking camera and facial expression recognition software.

[0381] Step 8: Optimizing Ad Display

[0382] The server selects the most suitable advertisement based on the analyzed emotion and displays it in the user's field of view. The input is the analyzed emotion data, and the output is the displayed advertisement image. Specifically, the server uses an advertisement selection algorithm and a display device.

[0383] An example of a prompt is, "Design an application that recognizes emotions in real time from the gaze and facial expressions captured by the camera when the user is wearing smart glasses, and displays the most appropriate advertisement based on that emotion. Specifically, implement a function that uses emotion recognition technology to detect the main emotion and selects and displays an appropriate advertisement image. In addition, the advertisement image will be selected from a list that has been pre-classified into several emotional categories."

[0384] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0385] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0386] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0387] [Second embodiment]

[0388] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0389] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0390] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0391] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0392] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0393] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0394] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0395] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0396] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0397] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0398] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0399] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0400] This invention is an AI-driven automation system aimed at improving business efficiency in companies. In the embodiment of the invention, a method for integrating and analyzing text, audio, image, and video data to automate business process optimization, document management, and customer support is described.

[0401] System Configuration

[0402] The system mainly consists of three components: a server, a terminal, and a user. The server is primarily responsible for data collection, preprocessing, analysis, optimization, management, and customer support. The terminal is a device that inputs and outputs data and provides a user interface. The user operates the system, uploads data, and reviews and approves proposal results.

[0403] Program processing overview

[0404] Data collection

[0405] The server periodically collects text, audio, image, and video data from various data sources, such as email servers, CRM systems, and customer support recording databases. Users use devices to upload data such as customer feedback and internal reports.

[0406] Data Preprocessing

[0407] The server preprocesses the collected data. For example, it transcribes audio data and standardizes the resolution of image data. It normalizes text data and removes unnecessary characters and whitespace. This ensures the data is in a consistent format and suitable for analysis.

[0408] Data analysis

[0409] The preprocessed data is then analyzed by the server. Natural language processing is used to perform semantic analysis of the text data, and sentiment analysis and topic models are built. Image and video data are also analyzed using computer vision technology to extract the necessary information. This allows issues within the business process to be identified.

[0410] Business process optimization

[0411] Based on the analysis results, the server generates business process optimization proposals. For example, if many customers report the same problem, it will identify the cause and propose improvements. Users can review the proposals on their devices and approve or modify them as necessary.

[0412] Document Management

[0413] The server uses an automatic document management system to efficiently manage internal and external documents. Uploaded documents are automatically categorized and placed in relevant folders. Users can easily access the documents they need using tag or keyword searches.

[0414] Automating customer interactions

[0415] The terminal receives inquiries from customers and immediately sends them to the server, which uses natural language understanding technology to analyze the inquiries and generate appropriate responses. Responses are automatically sent, and users can review and modify them as needed.

[0416] Specific examples

[0417] For example, if a company receives similar complaints from multiple customers at its support desk, the system works as follows:

[0418] 1. The server periodically collects new complaint emails from the support mailbox.

[0419] 2. The server preprocesses the email data and normalizes the text data.

[0420] 3. The server uses natural language processing to analyze the content of the complaint and extract key topics and frequently occurring keywords.

[0421] 4. Based on the analysis results, the server identifies where problems are concentrated in specific products or services and generates improvement proposals.

[0422] 5. The user checks the proposed improvements on the device and approves them if necessary.

[0423] 6. Complaint data will be automatically categorized and saved in relevant folders for future reference.

[0424] 7. When a new inquiry comes in from a customer, the server generates an automatic response and replies to the customer quickly.

[0425] By implementing this invention, companies can significantly improve their business efficiency and increase customer satisfaction.

[0426] The processing flow will be explained below.

[0427] Step 1:

[0428] The server collects data from various sources: email data from the mail server, customer interaction data downloaded from the CRM system, and audio data from the recording database.

[0429] Step 2:

[0430] Users upload data to the system using their devices. Specifically, they drag and drop customer feedback audio files and image data onto a dedicated upload interface to send them to the server.

[0431] Step 3:

[0432] The server preprocesses the collected data in bulk. For example, it uses a speech recognition API to convert voice data into text, an image conversion library to standardize image resolution, and normalizes the text data to remove unnecessary characters and spaces.

[0433] Step 4:

[0434] The server analyzes the preprocessed data. For text data, natural language processing (NLP) is used to perform sentiment analysis and build topic models. Specifically, Python NLP libraries (e.g., NLTK, spaCy) are used to analyze the meaning of the text. For image and video data, computer vision techniques are used, such as OpenCV and TensorFlow, to extract features for facial recognition and object detection.

[0435] Step 5:

[0436] The server generates business process optimization proposals based on the analysis results. For example, if there are many complaints about a particular product, it uses a rule engine to identify the cause and propose improvement measures.

[0437] Step 6:

[0438] The user checks the business process optimization proposals from the server through their terminal. Specifically, an interface is provided for reviewing the proposals and approving or adjusting them as necessary. If the user is satisfied with the proposals, they click the approval button to put them into action.

[0439] Step 7:

[0440] The server automatically classifies and manages documents. For example, it automatically analyzes uploaded documents such as contracts and customer feedback documents, attaches relevant tags, and moves them to the appropriate folders, allowing users to search more efficiently.

[0441] Step 8:

[0442] Users can search for and access the documents they need. Specifically, they can use keyword search and tag filter functions to easily find the documents they are looking for. They can then select the documents they need from the search results, access them, and obtain the information they need.

[0443] Step 9:

[0444] The terminal receives inquiries from customers, analyzes the content using natural language understanding (NLU) technology, and sends the inquiry content to the server.

[0445] Step 10:

[0446] The server generates an appropriate response based on the inquiry. Specifically, it searches for similar cases using past inquiry data and a knowledge base, and generates an automatic response. The generated response is automatically sent to the customer.

[0447] Step 11:

[0448] The user checks the automated response generated by the server, makes any necessary corrections, and then sends the final reply. Specifically, the user checks the response text on the terminal interface, corrects it, and then clicks the send button to provide accurate information to the customer.

[0449] By executing each processing step sequentially in this way, companies can achieve efficient data management and analysis, optimize automated business processes, and respond quickly and appropriately to customers.

[0450] Example 1

[0451] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0452] Modern companies are required to efficiently collect and analyze information from diverse data sources and quickly and accurately optimize their business processes. However, conventional systems require a great deal of time and effort to integrate and analyze data in different formats, and there are a lack of efficient ways to improve business processes. Furthermore, automation of document management and customer support is insufficient, resulting in increased effort. The purpose of this invention is to solve these problems.

[0453] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0454] In this invention, the server includes means for collecting text, audio, image, and video data from various data sources, means for preprocessing the collected data, transcribing it, standardizing its resolution, normalizing it, and converting it into a standard format, means for analyzing the preprocessed data and proposing business process optimization using natural language processing and computer vision technologies, means for performing automated document management, converting documents into text using OCR technology, and classifying them, and means for automating customer support using natural language understanding technology. This makes it possible to efficiently integrate and analyze data in different formats, quickly optimize business processes, and automate document management and customer support.

[0455] A "data source" is a source of information used by a system to gather information.

[0456] "Preprocessing" refers to a series of processes to format collected data so that it is easier to analyze.

[0457] "Transcription" is the process of converting audio data into text data.

[0458] "Resolution unification" is a process of unifying the resolution of image data to a certain standard.

[0459] "Normalization" is the process of removing unnecessary characters and spaces from text data and formatting it into a consistent format.

[0460] "Natural language processing" is a technology that allows computers to understand and analyze human language and extract meaning.

[0461] "Computer vision technology" is a technology that analyzes image and video data to extract necessary information.

[0462] "Business process optimization" is the process of proposing improvements and streamlining of business processes based on the results of analysis.

[0463] "Automated document management" is a system that automates processes such as document classification, storage, and retrieval.

[0464] "OCR technology" is an abbreviation for Optical Character Recognition, and is a technology that extracts character information from image data.

[0465] "Natural language understanding technology" is a technology that allows computers to understand human language and generate appropriate responses and actions.

[0466] "Customer response automation" is the process by which a system automatically generates and responds to customer inquiries.

[0467] This invention is an AI-driven automation system that collects text, audio, image, and video data from multiple data sources, and preprocesses and analyzes this data to optimize corporate business processes.

[0468] System Configuration

[0469] This system is mainly composed of three components: a server, a terminal, and a user. The server is primarily responsible for data collection, preprocessing, analysis, business process optimization, document management, and customer support. The terminal functions as the user interface, inputting and outputting data. The user operates the system, uploading data, and confirming and approving proposal results.

[0470] System action

[0471] Data collection

[0472] The server periodically collects text, audio, image, and video data from data sources such as email servers, CRM systems, and customer support recording databases. The specific software used includes data collection agents and API interfaces. Users use devices to upload data, such as customer feedback and internal reports, to the system.

[0473] Data Preprocessing

[0474] The server preprocesses the collected data. For audio data, it transcribes it using speech recognition software (e.g., speech recognition API). For image data, it uses image processing software (e.g., OpenCV) to standardize the resolution. For text data, it normalizes it using a natural language processing toolkit (e.g., NLTK) to remove unnecessary characters and spaces.

[0475] Data analysis

[0476] The preprocessed data is then analyzed by the server. Natural language processing tools (e.g., spaCy) are used to perform semantic analysis on text data, and sentiment analysis and topic models are built. Image and video data are analyzed using computer vision techniques (e.g., TensorFlow) to extract the necessary information. This allows issues within the business process to be identified.

[0477] Business process optimization

[0478] Based on the analysis results, the server generates business process optimization proposals. If many customers have reported similar problems, it will identify the cause and propose specific improvement measures (for example, updating FAQs or changing processes). Users can check the proposals on their devices and approve or modify them as necessary.

[0479] Document Management

[0480] The server uses an automated document management system (e.g., a document management platform) to efficiently manage internal and external documents. Uploaded documents are converted into text using OCR technology (e.g., optical character recognition tools), and automatically classified and stored. Users can easily access the documents they need using tag and keyword searches.

[0481] Automating customer interactions

[0482] The terminal receives inquiries from customers and immediately sends the details to the server. The server uses natural language understanding technology (e.g., language understanding models) to analyze the inquiry and automatically generate an appropriate response. Responses are sent automatically, but users can review and modify complex inquiries.

[0483] Specific examples

[0484] For example, if a company receives similar complaints from multiple customers at its support desk, the system works as follows:

[0485] 1. The server periodically collects new complaint emails from the support mailbox.

[0486] 2. The server preprocesses the email data and normalizes the text data.

[0487] 3. The server uses natural language processing to analyze the content of the complaint and extract key topics and frequently occurring keywords.

[0488] 4. Based on the analysis results, the server identifies where problems are concentrated in specific products or services and generates improvement proposals.

[0489] 5. The user checks the proposed improvements on the device and approves them if necessary.

[0490] 6. Complaint data will be automatically categorized and saved in relevant folders for future reference.

[0491] 7. When a new inquiry comes in from a customer, the server generates an automatic response and replies to the customer quickly.

[0492] Prompt Sentence Examples

[0493] Example of an input prompt for a generative AI model:

[0494] "Generate optimization proposals from the following workflow:

[0495] 1. Receiving and processing customer complaints

[0496] 2. Internal Quality Assurance Process

[0497] 3. Product Development Cycle

[0498] Please provide a summary of your analysis and specific suggestions for improvement.

[0499] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0500] Step 1: Data collection

[0501] The server periodically collects text, voice, image, and video data from data sources such as email servers, CRM systems, and customer support recording databases. The server retrieves this data through an API interface or by using a data collection agent and stores it in data storage. Users use their devices to upload data such as customer feedback and internal reports and send it to the server. The input of this step is raw data such as emails, voice recordings, and images, and the output is the collected, unprocessed data.

[0502] Step 2: Data Preprocessing

[0503] The server preprocesses the collected raw data. For audio data, it converts it into text using speech recognition software (e.g., a speech recognition API). For image data, it uses image processing software (e.g., OpenCV) to standardize the resolution and perform noise reduction if necessary. Furthermore, it normalizes the text data using a natural language processing toolkit (e.g., NLTK) to remove unnecessary characters and spaces. The input of this step is the raw data, and the output is the preprocessed data.

[0504] Step 3: Data analysis

[0505] The server analyzes the preprocessed data. Text data is semantically analyzed using natural language processing tools (e.g., spaCy) to perform sentiment analysis and build topic models. Image data and video data are analyzed using computer vision techniques (e.g., TensorFlow) to extract the necessary information. The analysis results include the data's topics, sentiment scores, and the detection of specific events and objects. The input to this step is the preprocessed data, and the output is the analysis results.

[0506] Step 4: Optimize business processes

[0507] The server generates optimization proposals for business processes based on the results of data analysis. For example, if many complaints with low sentiment scores are concentrated around a specific product, the server identifies the cause and proposes countermeasures (e.g., FAQ updates, product modifications). The user can review these proposals on their device and approve or modify them as necessary. The input to this step is the analysis results, and the output is optimization proposals.

[0508] Step 5: Document Management

[0509] The server uses an automated document management system (e.g., a document management platform) to efficiently manage internal and external documents. It converts uploaded documents into text using OCR technology (e.g., optical character recognition tools), classifies them based on categories, and stores them in relevant folders. Users can access the documents they need using a search interface. The input of this step is document data, and the output is classified and stored documents.

[0510] Step 6: Automate customer interactions

[0511] The terminal receives customer inquiries in real time and immediately sends the content to the server. The server uses natural language understanding technology (e.g., a language understanding model) to analyze the inquiry and generate an appropriate automated response. For simple inquiries, an automated response is sent to the customer immediately, but for complex inquiries, the user can review and modify the response. The input to this step is the text of the customer inquiry, and the output is the generated response.

[0512] (Application example 1)

[0513] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0514] In manufacturing processes within factories, the collection and analysis of a wide variety of data (text, audio, images, video) is not being carried out efficiently, making it difficult to optimize business processes and not fully automating document management and customer support. This makes it difficult to improve business efficiency and customer satisfaction. To solve this problem, an automated system that uses integrated data management and advanced analysis technology is required.

[0515] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0516] In this invention, the server includes a means for collecting text, audio, image, and video data, a means for preprocessing the collected data and converting it into a standard format, and a means for analyzing the preprocessed data and proposing business process optimization. This enables data analysis and optimization proposals to be made using a clustering algorithm in a manufacturing environment. The server also includes a means for converting audio data into text data and analyzing it, and a means for collecting and preprocessing data from various sensors, cameras, and audio recorders. Furthermore, by incorporating a means for analyzing the sentiment of audio data using natural language processing technology and improving the efficiency of customer service, it is possible to improve not only business efficiency but also customer satisfaction.

[0517] "Text data" refers to character string information written in a natural language, and includes formats such as documents, emails, and reports.

[0518] "Voice data" refers to digitally recorded human speech or acoustic signals, and is used for speech recognition and emotion analysis.

[0519] "Image data" is visual information stored in digital form, including photographs, graphics, and diagrams.

[0520] "Moving image data" refers to data that digitally records visual information that changes over time, including video clips and video recordings.

[0521] "Preprocessing" refers to a series of operations to convert collected data into a format suitable for analysis, including format standardization and noise removal.

[0522] A "clustering algorithm" is a machine learning technique for classifying data into multiple clusters, based on patterns and similarities in the data.

[0523] "Natural language processing" is a technology that enables computers to understand and analyze human language, and is used for semantic and sentiment analysis of text.

[0524] "Sentiment analysis" is a technology that detects and classifies emotions from text and voice data, and is useful for automating customer responses.

[0525] A "sensor" is a device that detects the physical state of an environment or object and is used to collect data.

[0526] A "camera" is a device that captures visual information and stores it as digital data, and is used to collect image data and video data.

[0527] "Audio recording device" means a device for recording audio in digital form and is used to collect speech and acoustic data.

[0528] This invention is an AI-driven automation system that aims to improve the efficiency of work processes in a factory. In an embodiment of the invention, the system is composed of three entities: a server, a terminal, and a user.

[0529] System Configuration

[0530] The server plays the primary role of collecting data, preprocessing, analyzing, optimizing, managing, and responding to customers. The terminal is a device that inputs and outputs data and provides a user interface, and is also used for robots installed on production lines in factories. The user operates the system, uploads data, and checks and approves proposed results.

[0531] Program processing overview

[0532] Data collection

[0533] The server periodically collects text data, audio data, image data, and video data from various sensors, cameras, and audio recorders in the factory. For example, the server acquires this data from the production line's recording devices and quality control systems. Users use their terminals to upload data such as production daily reports and error logs.

[0534] Data Preprocessing

[0535] The server preprocesses the collected data. For example, it converts audio data to text data and standardizes the resolution of image data. Text data is normalized and unnecessary characters and spaces are removed. This ensures that the data is in a consistent format and suitable for analysis.

[0536] Data analysis

[0537] The preprocessed data is then analyzed by the server. Natural language processing is used to perform semantic and sentiment analysis of the text data and build a topic model. A clustering algorithm is also used to classify the manufacturing data into clusters. Image and video data is analyzed using computer vision technology to extract the necessary information. This allows issues in the manufacturing process within the factory to be identified.

[0538] Business process optimization

[0539] Based on the analysis results, the server generates business process optimization proposals. For example, if multiple manufacturing errors occur for the same reason, the server will identify the cause and propose improvement measures. Users can check the proposals on their devices and approve or modify them as necessary.

[0540] Document Management

[0541] The server uses an automatic document management system to efficiently manage documents inside and outside the factory. Uploaded documents are automatically categorized and placed in relevant folders. Users can easily access the documents they need using tag or keyword searches.

[0542] Automating customer interactions

[0543] The terminal receives inquiries from customers and immediately sends them to the server, which uses natural language understanding technology to analyze the inquiries and generate appropriate responses. Responses are automatically sent, and users can review and modify them as needed.

[0544] Specific examples

[0545] As a concrete example, when quality control is performed on a manufacturing line in a factory, the system operates as follows.

[0546] 1. The server periodically collects new data from the production line's sensor data and quality control records.

[0547] 2. The server preprocesses these data and converts them into a standard format.

[0548] 3. The server analyzes the data using natural language processing and clustering algorithms to identify key manufacturing challenges and frequent problems.

[0549] 4. Based on the analysis results, the server identifies where problems are concentrated in specific manufacturing processes and generates improvement proposals.

[0550] 5. The user checks the proposed improvements on the device and approves them if necessary.

[0551] 6. Manufacturing data is automatically categorized and saved in relevant folders for future reference.

[0552] Prompt Sentence Examples

[0553] "Collect new complaint emails from the support mailbox, analyze the content of the complaint using natural language processing, and generate a report of the results."

[0554] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0555] Step 1:

[0556] Data collection

[0557] The server periodically collects text data, voice data, image data, and video data from various sensors, cameras, and audio recorders in the factory. For example, the server receives inputs from the production line monitoring system and quality control system to acquire these data, and outputs the collected data to be stored in the centralized management system.

[0558] Step 2:

[0559] Data Preprocessing

[0560] The server preprocesses the collected data. For example, it converts audio data into text data and standardizes the resolution of image data. Specifically, it uses voice recognition software to convert audio data into text and image processing algorithms to standardize the resolution of images. This preprocessing ensures that the data is in a consistent format, providing appropriate input data for analysis. The output is standardized text, audio, image, and video data.

[0561] Step 3:

[0562] Data analysis

[0563] The preprocessed data is then analyzed by the server, which uses natural language processing to perform semantic and sentiment analysis of the text data. For example, a generative AI model is used to detect sentiment from the text data and build a topic model. A clustering algorithm is also used to classify the manufacturing data into clusters. Image and video data is analyzed using computer vision techniques to extract information for anomaly detection and quality assessment on the production line. The output of this step is a detailed report based on the analysis results.

[0564] Step 4:

[0565] Business process optimization

[0566] Based on the analysis results, the server generates optimization proposals for business processes. The server identifies patterns of manufacturing errors and frequent problems and proposes specific improvement measures. It also uses a customized generative AI model to generate optimization proposals. These optimization proposals include specific instructions and action plans for improving the manufacturing process, thereby enabling efficient business operations. The output is a proposal for improving the business process.

[0567] Step 5:

[0568] Document Management

[0569] The server uses an automatic document management system to efficiently manage documents inside and outside the factory. Uploaded documents are automatically categorized and placed in relevant folders. They are also designed to be easily accessible using tag and keyword searches, allowing users to quickly access the documents they need. This improves the efficiency of document management. The output is categorized and organized document data.

[0570] Step 6:

[0571] Automating customer interactions

[0572] The terminal receives an inquiry from a customer and immediately sends the content to the server. The server uses natural language understanding technology to analyze the content of the inquiry and generate an appropriate response. The generated response is sent automatically, but the user can confirm and modify it if necessary. A prompt sentence (e.g., "Please collect new complaint emails from the support mailbox, analyze the content of the complaint using natural language processing, and generate a report of the results") may also be used to generate an initial response. The output of this step is a prompt and appropriate response to improve customer satisfaction.

[0573] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0574] This invention is an AI-driven automation system aimed at improving the efficiency of corporate operations, and in particular, by combining it with an emotion engine that recognizes user emotions, it achieves more advanced optimization of business processes and automation of customer support. In the embodiments of the present invention, a method is described for integrating and analyzing text, audio, image, and video data to automate optimization of business processes, document management, and customer support that take user emotions into consideration.

[0575] System Configuration

[0576] The system mainly consists of three components: a server, a terminal, and a user. The server is primarily responsible for data collection, preprocessing, analysis, optimization, management, customer service, and emotion recognition. The terminal is a device that inputs and outputs data and provides a user interface. The user operates the system, uploads data, and reviews and approves proposal results.

[0577] Program processing overview

[0578] Data collection

[0579] The server collects data from various sources, including email servers, CRM systems, and customer support recording databases, periodically retrieving text, audio, image, and video data. Users use their devices to upload data, such as customer feedback and internal reports, to the system.

[0580] Data Preprocessing

[0581] The server preprocesses the collected data and converts it into a standard format. For example, audio data is transcribed, image data is standardized, and text data is normalized to remove unnecessary characters and whitespace. This ensures the data is in a consistent format and suitable for analysis.

[0582] Data Analysis and Emotion Recognition

[0583] The preprocessed data is then analyzed by the server. Natural language processing is used to perform semantic analysis of the text data, and sentiment analysis is also performed simultaneously. For example, positive, negative, and neutral emotions are extracted from the text data. Voice data is also analyzed using a sentiment extraction engine to recognize the customer's emotions. Image and video data are also analyzed using computer vision technology to identify emotions from the user's facial expressions.

[0584] Business process optimization

[0585] Based on the analysis results and the recognized emotions, the server generates business process optimization proposals. For example, if many customers have negative emotions about a particular product, the server will identify the cause and propose improvements. The user can review the proposals on their device and approve or modify them as necessary.

[0586] Document Management

[0587] The server uses an automatic document management system to efficiently manage internal and external documents. Uploaded documents are automatically categorized and placed in relevant folders. Users can easily access the documents they need using tag or keyword searches.

[0588] Automating customer interactions

[0589] The terminal receives inquiries from customers and immediately sends them to the server. The server uses natural language understanding technology to analyze the inquiries and generate appropriate responses. Responses can also be tailored based on the user's sentiment. Responses are automatically sent, and users can review and modify them as needed.

[0590] Specific examples

[0591] For example, if a company receives similar complaints from multiple customers at its support desk, the system works as follows:

[0592] 1. The server periodically collects new complaint emails from the support mailbox.

[0593] 2. The server preprocesses the email data and normalizes the text data.

[0594] 3. The server uses natural language processing to analyze the content of the complaint and extract key topics and frequently occurring keywords.

[0595] 4. At the same time, the server uses an emotion engine to recognize customer emotions from the text data. For example, if many emails express negative emotions, this is detected.

[0596] 5. Based on the analysis results and sentiment data, the server identifies problems concentrated in specific products or services and generates improvement proposals. For example, if there is a lot of negative feedback stating that "there are problems with the quality of a specific product," it will propose improvement measures for that product.

[0597] 6. The user checks the proposed improvements on the terminal and approves them if necessary.

[0598] 7. Complaint data will be automatically categorized and saved in relevant folders for future reference.

[0599] 8. When a new inquiry comes in from a customer, the server generates an automatic response and quickly replies to the customer with a message that reflects the user's feelings.

[0600] By implementing this invention, enterprises can significantly improve their business efficiency and provide high-quality customer service that takes into consideration users' emotions.

[0601] The processing flow will be explained below.

[0602] Step 1:

[0603] The server collects data from various sources: email data from the mail server, customer interaction data from the CRM system, and voice data from the recording database. The server periodically checks these data sources and collects any new data.

[0604] Step 2:

[0605] Users upload data to the system using their devices. Specifically, they drag and drop customer feedback audio files and image data into a dedicated upload interface and send them to the server. Users do this as part of their daily work.

[0606] Step 3:

[0607] The server preprocesses the collected data. For example, it transcribes audio data and standardizes the resolution of image data. It normalizes text data and removes unnecessary characters and whitespace. This preprocessing ensures that the data is in a consistent format and suitable for analysis.

[0608] Step 4:

[0609] The server analyzes the preprocessed data. Specifically, it uses natural language processing (NLP) to perform semantic analysis of the text data. It uses Python NLP libraries (e.g., spaCy, NLTK) to analyze the structure of the text and build sentiment analysis and topic models. It also uses speech recognition and emotion recognition engines to extract emotions from audio data.

[0610] Step 5:

[0611] The server analyzes image and video data. It uses computer vision technology (e.g., OpenCV, TensorFlow) to analyze the image data and identify emotions from the user's facial expressions. Similarly, it extracts emotions from video data based on specific facial expressions and movements.

[0612] Step 6:

[0613] The server uses an emotion recognition engine to extract the user's emotion from the analyzed data. For text data, it calculates an emotion score such as positive, negative, or neutral, and generates similar emotion scores for audio and image data.

[0614] Step 7:

[0615] The server generates business process optimization proposals based on the analysis results and sentiment data. For example, if many customers have negative feelings about a particular product, the server identifies the cause and proposes improvement measures. The proposals are generated in the form of specific action plans and changes.

[0616] Step 8:

[0617] The user checks the business process optimization proposals from the server through their terminal. For example, they review the proposals displayed on the terminal and approve or adjust them as necessary. If the user is satisfied with the proposals, they click the approval button to put them into action.

[0618] Step 9:

[0619] The server automatically classifies and manages documents. For example, it automatically analyzes uploaded documents such as contracts and customer feedback documents, attaches relevant tags, and moves them to the appropriate folders, improving document search efficiency.

[0620] Step 10:

[0621] Users search for and access the documents they need. Specifically, they use the device's search function to enter keywords or tags to quickly identify the desired document. They then select the relevant document from the search results and access it to obtain the information they need.

[0622] Step 11:

[0623] The terminal receives inquiries from customers, analyzes the content using natural language understanding (NLU) technology, and sends the inquiry to a server. For example, this is done by chatbots and email reception interfaces.

[0624] Step 12:

[0625] The server generates an appropriate response based on the inquiry. Specifically, it searches for similar cases using past inquiry data and a knowledge base, and generates an automatic response. The generated response is quickly sent to the customer.

[0626] Step 13:

[0627] The user checks the automated response generated by the server, makes any necessary corrections, and then sends the final reply. For example, by checking the response text on the terminal interface, correcting it, and then clicking the send button, the user can provide accurate information to the customer.

[0628] Example 2

[0629] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0630] Modern business activities require the efficient management and analysis of massive amounts of data, as well as the optimization of business processes and the automation of customer responses. In particular, an approach that handles text, voice, image, and video data in an integrated manner and takes user emotions into account is essential, but there are currently no systems that can effectively achieve this. Furthermore, managing and analyzing data manually is time-consuming, costly, and prone to human error. Therefore, there is a need for a comprehensive system that automates everything from data collection and analysis to optimization proposals, document management, and customer responses.

[0631] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0632] In this invention, the server includes means for collecting text, audio, image, and video data from various data sources, means for preprocessing the collected data and converting it into a standard format, and means for analyzing the preprocessed data using a natural language processing and emotion recognition engine. This enables companies to efficiently manage and analyze massive amounts of data, optimize business processes, and automate customer responses.

[0633] "Various data sources" refers to multiple information sources, including email servers, customer relationship management systems, customer support recording databases, etc.

[0634] "Text data" refers to data expressed in text format, such as emails, reports, and chat history.

[0635] "Audio data" refers to data expressed in the form of voice, such as audio files and recorded data.

[0636] "Image data" refers to data expressed in a visual format, such as photographs, screenshots, and illustrations.

[0637] "Video data" refers to data that is presented in a moving visual format, such as a video clip or recorded data.

[0638] "Collect" refers to the act of obtaining data and putting it into a system.

[0639] "Preprocessing" refers to the initial processing of collected data to convert it into a form that is easier to analyze. Examples include transcribing audio data and normalizing text data.

[0640] "Standard format" refers to a standard for standardizing data into a consistent format.

[0641] "Natural language processing" refers to the technology that enables computers to understand and analyze natural language written and spoken by humans.

[0642] An "emotion recognition engine" refers to software or algorithms that identify and analyze emotions contained within data.

[0643] "Analyzing" refers to the act of analyzing data to understand its content and patterns and extract information.

[0644] "Business process optimization" refers to the act of planning and implementing proposals and measures to improve the efficiency of corporate activities.

[0645] "Automated document management" refers to the automatic performance of management tasks such as document classification, storage, and retrieval by a computer system.

[0646] "Customer response automation" refers to a system automatically generating and sending appropriate responses to customer inquiries.

[0647] This invention is an AI-driven automation system aimed at improving business efficiency, and in particular, by combining an emotion engine that recognizes user emotions, it optimizes business processes and automates customer support. This system is mainly composed of three components: a server, a terminal, and a user. Specific embodiments of this system are described below.

[0648] System Configuration

[0649] The system consists of three components: a server, a terminal, and a user. The server is primarily responsible for data collection, preprocessing, analysis, optimization, management, customer support, and emotion recognition. The terminal is a device that inputs and outputs data and provides a user interface. The user operates the system, uploads data, and confirms and approves proposal results. Specific implementation examples of each component are shown below.

[0650] server

[0651] The server periodically collects text, audio, image, and video data from various data sources (e.g., email servers, CRM systems, customer support recording databases, etc.). The collected data is ingested through an ETL (Extract, Transform, Load) process. The server then preprocesses the collected data. Audio data is transcribed using a speech-to-text engine (e.g., Google Cloud Speech-to-Text), image data is standardized in resolution, and video data is broken down into frames. Text data is normalized, and unnecessary characters and whitespace are removed. Regular expressions and natural language processing tools (e.g., NLTK) are used in this process.

[0652] The preprocessed data is then analyzed using a natural language processing (NLP) engine (e.g., BERT or GPT model) to perform semantic analysis of the text data. A sentiment analysis engine (e.g., Sentiment140) is used to extract positive, negative, or neutral sentiment. Audio data is also analyzed using a sentiment extraction engine. Finally, image and video data are analyzed using computer vision techniques (e.g., OpenCV or deep learning models) to identify emotions from the user's facial expressions.

[0653] Based on the analysis results, the server generates business process optimization proposals. For example, if many customers have negative feelings about a particular product, it will suggest improvements for that product. It also uses an automated document management system to efficiently manage uploaded documents. Documents are categorized into appropriate folders, and necessary documents can be quickly accessed using tag or keyword searches.

[0654] Terminal

[0655] The terminal is a device that allows users to input and output data and provides a user interface. Users upload data through the terminal and review and approve proposals and analysis results. When users upload data, the terminal checks the data format and displays a warning if the format is inappropriate. The terminal also receives inquiries from customers and immediately sends the details to the server.

[0656] User

[0657] Users operate the system to upload data and review and approve proposal results. Specifically, they use terminals to upload data such as customer feedback and internal reports to the system. They also review the proposal content and approve or modify it if necessary. When a customer inquiry is received, they can also review the generated response and make modifications as necessary.

[0658] Specific examples

[0659] For example, if a company receives similar complaints from multiple customers at its support desk, the system works as follows:

[0660] 1. The server periodically collects new complaint emails from the support mailbox.

[0661] 2. The server preprocesses the email data and normalizes the text data.

[0662] 3. The server uses natural language processing to analyze the content of the complaint and extract key topics and frequently occurring keywords.

[0663] 4. At the same time, the server uses an emotion engine to recognize customer emotions from the text data. For example, if many emails express negative emotions, this is detected.

[0664] 5. Based on the analysis results and sentiment data, the server identifies problems concentrated in specific products or services and generates improvement proposals. For example, if there is a lot of negative feedback stating that "there are problems with the quality of a specific product," it will propose improvement measures for that product.

[0665] 6. The user checks the proposed improvements on the terminal and approves them if necessary.

[0666] 7. Complaint data will be automatically categorized and saved in relevant folders for future reference.

[0667] 8. When a new inquiry comes in from a customer, the server generates an automatic response and quickly replies to the customer with a message that reflects the user's feelings.

[0668] Prompt Sentence Examples

[0669] Below are some examples of text data to analyze and specific prompts to extract sentiment:

[0670] Prompt: "Conduct a customer feedback analysis and generate a report that extracts key topics and their respective sentiments. Analyze the following feedback: 'Product X is difficult to use. Support is slow to respond.'"

[0671] By feeding this prompt into a generative AI model, the system can analyze the main topics of the feedback (e.g., usability, support response) and sentiment (negative) and generate a corresponding report.

[0672] In this way, the present invention significantly improves the business efficiency of companies, and in particular enables high-quality responses that take into account the user's emotions.

[0673] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0674] Step 1:

[0675] The server collects text, audio, image, and video data from various data sources (e.g., email servers, CRM systems, customer support recording databases, etc.) and periodically ingests new data by running an ETL (Extract, Transform, Load) process. This step uses data from various data sources as input and produces the collected data as output.

[0676] Step 2:

[0677] The server preprocesses the collected data. For example, audio data is transcribed using a speech-to-text engine, image data is standardized in resolution, and video data is broken down into frames. Text data is normalized using regular expressions and natural language processing tools. This step uses the collected raw data as input and produces preprocessed data as output.

[0678] Step 3:

[0679] The server analyzes the preprocessed data. It uses a natural language processing engine (e.g., a GPT model) to perform semantic analysis of the text data and a sentiment analysis engine to extract positive, negative, or neutral sentiment. Audio data is also analyzed using a sentiment extraction engine, and image and video data are analyzed using computer vision techniques. This step uses the preprocessed data as input and generates analysis results as output.

[0680] Step 4:

[0681] The server generates business process optimization proposals based on the analysis results. For example, if a particular product has received a lot of negative feedback, it generates improvement proposals for that product. It also includes automated processes for customer support. This step uses the analysis results as input and generates optimization proposals and automated responses as output.

[0682] Step 5:

[0683] The user uses a terminal to review the improvement proposals and the automated response. The terminal displays the proposals and provides an interface for the user to approve or modify them as needed. If the user makes modifications, the modified proposals are fed back into the system. This step uses the optimization proposals and automated response as inputs and generates the proposals that have been reviewed and modified by the user as output.

[0684] Step 6:

[0685] The server uses an automated document management system to classify and store the uploaded documents. The documents are categorized into appropriate folders and can be efficiently accessed using tag and keyword searches. This step uses the uploaded documents as input and produces classified and stored documents as output.

[0686] Step 7:

[0687] When the terminal receives a customer inquiry, it immediately sends the content to the server, which analyzes the inquiry and generates an appropriate response using natural language understanding technology. The response can also be tailored based on the user's sentiment. This step uses the customer inquiry information as input and provides the generated response as output.

[0688] (Application example 2)

[0689] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0690] In conventional corporate business processes, it was difficult to analyze user emotions in real time and provide optimal responses. Furthermore, when it came to displaying advertisements, static ads were often displayed without considering user emotions, making it impossible to deliver advertisements effectively. This resulted in problems such as lower customer satisfaction and reduced business efficiency.

[0691] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0692] In this invention, the server includes means for collecting text, audio, image, and video data, means for preprocessing the collected data and converting it into a standard format, means for analyzing the preprocessed data and proposing business process optimization, means for automated document management, means for automating customer support, means for tracking user gaze and analyzing emotions in real time, and means for displaying optimal advertisements based on the analyzed emotions. This allows business processes and advertisement display to be optimized based on user emotions, enabling highly efficient and high-quality customer support.

[0693] "Text data" refers to data that contains sentences or character information, and includes emails, chat messages, documents, and the like.

[0694] "Voice data" means data obtained by recording a human voice or other sounds, including voice messages, telephone recordings, and customer support audio recordings.

[0695] "Image data" refers to visual information recorded as still images, including photographs, screenshots, illustrations, etc.

[0696] "Movie data" refers to dynamic visual information consisting of a series of image frames, including video clips and video recordings.

[0697] "Collection methods" are the technical means used to obtain text, audio, image, and video data from a variety of sources, including database access and use of APIs.

[0698] "Preprocessing means" refers to technology that converts collected data into a format that is easier to analyze, such as transcribing audio data or standardizing image resolution.

[0699] A "standard format" is a format in which data of different formats is converted into a consistent format that can be analyzed, making data analysis easier.

[0700] "Means of analysis" refers to technologies that use preprocessed data to extract information and optimize or propose business processes, including machine learning and natural language processing.

[0701] "Means for proposing optimization" refers to techniques that propose methods and means for improving the efficiency of business processes based on the analysis results, and includes formulating plans and presenting improvement measures.

[0702] "Means for document management" refers to technology for efficiently managing documents, including automatic classification and tagging of documents, and search functions.

[0703] "Means for automating customer support" refers to technology that automatically responds to inquiries and requests from customers, and includes chatbots and automatic reply systems.

[0704] "Gaze tracking means" refers to technology that detects the direction of a user's gaze and monitors it in real time, and includes gaze tracking cameras and sensors.

[0705] "Means for analyzing emotions in real time" refers to technology that detects the user's emotions at that time from their facial expressions, voice, and text, and includes facial expression recognition engines and emotion analysis algorithms.

[0706] The "means for displaying advertisements" refers to technology that displays appropriate advertisements to users based on the analyzed emotions, and includes displays and marketing engines.

[0707] This invention is an AI-driven automation system aimed at improving business efficiency and optimizing customer service. The system includes means for collecting text, voice, image, and video data, means for preprocessing the collected data and converting it into a standard format, means for analyzing the preprocessed data and proposing business process optimization, means for automated document management, and means for automating customer service. It also includes means for tracking a user's gaze and analyzing their emotions in real time, and means for displaying optimal advertisements based on the analyzed emotions.

[0708] System Configuration

[0709] This system is mainly composed of three components: a server, a terminal, and a user. The server is primarily responsible for data collection, preprocessing, analysis, optimization, emotion recognition, and display of advertisements. The terminal is a device that inputs and outputs data and provides a user interface. The user operates the system, uploads data, and reviews and approves the proposed results.

[0710] Data collection

[0711] The server collects text, audio, image, and video data from various data sources, such as email servers, CRM systems, and customer support recording databases, and users use their devices to upload data to the system, including customer feedback and internal reports.

[0712] Data Preprocessing

[0713] The server preprocesses the collected data and converts it into a standard format. For example, audio data is transcribed, image data is standardized, and text data is normalized to remove unnecessary characters and whitespace. This ensures the data is in a consistent format and suitable for analysis.

[0714] Data Analysis and Emotion Recognition

[0715] The preprocessed data is then analyzed by the server. Natural language processing is used to perform semantic analysis of the text data, and sentiment analysis is also performed simultaneously. For example, positive, negative, and neutral emotions are extracted from the text data. Voice data is also analyzed using a sentiment extraction engine to recognize customer emotions. Image and video data are also analyzed using computer vision technology to identify emotions from the user's facial expressions.

[0716] Business process optimization

[0717] Based on the analysis results and the recognized emotions, the server generates business process optimization proposals. For example, if many customers have negative emotions about a particular product, the server will identify the cause and propose improvements. The user can review the proposals on their device and approve or modify them as necessary.

[0718] Document Management

[0719] The server efficiently manages internal and external documents using an automatic document management system. Uploaded documents are automatically categorized and placed in relevant folders. Users can easily access the documents they need using tag or keyword searches.

[0720] Automating customer interactions

[0721] The terminal receives inquiries from customers and immediately sends them to the server. The server uses natural language understanding technology to analyze the inquiries and generate appropriate responses. It can also adjust the responses based on the user's sentiment. Responses are automatically sent, and users can review and modify them as needed.

[0722] Eye Tracking and Emotion Analysis

[0723] When a user wears smart glasses, the eye tracking camera detects the direction of their gaze and analyzes their facial expressions in real time. The server collects this data and performs emotion analysis. Based on the analyzed emotion, the server selects the most appropriate advertisement and displays it in the user's field of view.

[0724] Specific examples

[0725] When a user wears smart glasses and browses web content, Serve analyzes their gaze and facial expressions in real time. For example, if a user shows interest in a particular product page, Serve can detect positive emotions and display relevant ads based on those emotions. This maximizes the effectiveness of advertising and improves the user experience.

[0726] An example of a prompt is, "Design an application that recognizes emotions in real time from the gaze and facial expressions captured by the camera when the user is wearing smart glasses, and displays the most appropriate advertisement based on that emotion. Specifically, implement a function that uses emotion recognition technology to detect the main emotion and selects and displays an appropriate advertisement image. In addition, the advertisement image will be selected from a list that has been pre-classified into several emotional categories."

[0727] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0728] Step 1: Data collection

[0729] The server periodically retrieves text, audio, image, and video data from mail servers, CRM systems, and customer support recording databases. Users use their devices to upload data, such as customer feedback and internal reports, to the system. The input is various data from external data sources, and the output is the collected raw data. Specifically, the server retrieves data using API calls and database queries.

[0730] Step 2: Preprocessing the data

[0731] The server preprocesses the collected data and converts it into a standard format. Specifically, it transcribes audio data, standardizes the resolution of image data, and normalizes text data to remove unnecessary characters and spaces. The input is the raw data collected, and the output is the preprocessed data. The server performs these conversions using a speech recognition engine and image processing algorithms.

[0732] Step 3: Data analysis and emotion recognition

[0733] The server analyzes the preprocessed data and uses natural language processing to perform semantic analysis of the text data, while simultaneously performing sentiment analysis. For example, positive, negative, and neutral emotions are extracted from the text data. Voice data is analyzed using an emotion extraction engine to recognize customer emotions. Image data and video data are analyzed using computer vision technology. The input is the preprocessed data, and the output is the analysis results and emotional data. Specifically, the server performs these analyses using a natural language processing library and a facial recognition engine.

[0734] Step 4: Optimize business processes

[0735] The server generates optimization proposals for business processes based on the analysis results and the recognized emotions. For example, if many customers have negative emotions about a particular product, the cause can be identified and improvement measures proposed. The input is the analysis results and emotion data, and the output is optimization proposals for business processes. Specifically, the server generates optimization proposals using machine learning algorithms.

[0736] Step 5: Document Management

[0737] The server uses an automated document management system to efficiently manage internal and external documents. Uploaded documents are automatically categorized and placed in relevant folders. Users can access the documents they need using tag or keyword searches. The input is the uploaded document data, and the output is organized and categorized documents. Specifically, the server uses document management software.

[0738] Step 6: Automate customer interactions

[0739] The terminal receives an inquiry from a customer and immediately sends the content to the server. The server uses natural language understanding technology to analyze the inquiry and generate an appropriate response. The input is the customer's inquiry, and the output is the generated response message. Specifically, the server uses a natural language understanding engine.

[0740] Step 7: Gaze Tracking and Emotion Analysis

[0741] When a user wears smart glasses, an eye-tracking camera is used to detect the direction of gaze and analyze facial expressions in real time. The server collects this data and performs emotion analysis. The input is the user's gaze and facial expression data, and the output is analyzed emotion data. Specifically, the server uses an eye-tracking camera and facial expression recognition software.

[0742] Step 8: Optimizing Ad Display

[0743] The server selects the most suitable advertisement based on the analyzed emotion and displays it in the user's field of view. The input is the analyzed emotion data, and the output is the displayed advertisement image. Specifically, the server uses an advertisement selection algorithm and a display device.

[0744] An example of a prompt is, "Design an application that recognizes emotions in real time from the gaze and facial expressions captured by the camera when the user is wearing smart glasses, and displays the most appropriate advertisement based on that emotion. Specifically, implement a function that uses emotion recognition technology to detect the main emotion and selects and displays an appropriate advertisement image. In addition, the advertisement image will be selected from a list that has been pre-classified into several emotional categories."

[0745] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0746] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0747] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0748] [Third embodiment]

[0749] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0750] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0751] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0752] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0753] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0754] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0755] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0756] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0757] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0758] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0759] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0760] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0761] This invention is an AI-driven automation system aimed at improving business efficiency in companies. In the embodiment of the invention, a method for integrating and analyzing text, audio, image, and video data to automate business process optimization, document management, and customer support is described.

[0762] System Configuration

[0763] The system mainly consists of three components: a server, a terminal, and a user. The server is primarily responsible for data collection, preprocessing, analysis, optimization, management, and customer support. The terminal is a device that inputs and outputs data and provides a user interface. The user operates the system, uploads data, and reviews and approves proposal results.

[0764] Program processing overview

[0765] Data collection

[0766] The server periodically collects text, audio, image, and video data from various data sources, such as email servers, CRM systems, and customer support recording databases. Users use devices to upload data such as customer feedback and internal reports.

[0767] Data Preprocessing

[0768] The server preprocesses the collected data. For example, it transcribes audio data and standardizes the resolution of image data. It normalizes text data and removes unnecessary characters and whitespace. This ensures the data is in a consistent format and suitable for analysis.

[0769] Data analysis

[0770] The preprocessed data is then analyzed by the server. Natural language processing is used to perform semantic analysis of the text data, and sentiment analysis and topic models are built. Image and video data are also analyzed using computer vision technology to extract the necessary information. This allows issues within the business process to be identified.

[0771] Business process optimization

[0772] Based on the analysis results, the server generates business process optimization proposals. For example, if many customers report the same problem, it will identify the cause and propose improvements. Users can review the proposals on their devices and approve or modify them as necessary.

[0773] Document Management

[0774] The server uses an automatic document management system to efficiently manage internal and external documents. Uploaded documents are automatically categorized and placed in relevant folders. Users can easily access the documents they need using tag or keyword searches.

[0775] Automating customer interactions

[0776] The terminal receives inquiries from customers and immediately sends them to the server, which uses natural language understanding technology to analyze the inquiries and generate appropriate responses. Responses are automatically sent, and users can review and modify them as needed.

[0777] Specific examples

[0778] For example, if a company receives similar complaints from multiple customers at its support desk, the system works as follows:

[0779] 1. The server periodically collects new complaint emails from the support mailbox.

[0780] 2. The server preprocesses the email data and normalizes the text data.

[0781] 3. The server uses natural language processing to analyze the content of the complaint and extract key topics and frequently occurring keywords.

[0782] 4. Based on the analysis results, the server identifies where problems are concentrated in specific products or services and generates improvement proposals.

[0783] 5. The user checks the proposed improvements on the device and approves them if necessary.

[0784] 6. Complaint data will be automatically categorized and saved in relevant folders for future reference.

[0785] 7. When a new inquiry comes in from a customer, the server generates an automatic response and replies to the customer quickly.

[0786] By implementing this invention, companies can significantly improve their business efficiency and increase customer satisfaction.

[0787] The processing flow will be explained below.

[0788] Step 1:

[0789] The server collects data from various sources: email data from the mail server, customer interaction data downloaded from the CRM system, and audio data from the recording database.

[0790] Step 2:

[0791] Users upload data to the system using their devices. Specifically, they drag and drop customer feedback audio files and image data onto a dedicated upload interface to send them to the server.

[0792] Step 3:

[0793] The server preprocesses the collected data in bulk. For example, it uses a speech recognition API to convert voice data into text, an image conversion library to standardize image resolution, and normalizes the text data to remove unnecessary characters and spaces.

[0794] Step 4:

[0795] The server analyzes the preprocessed data. For text data, natural language processing (NLP) is used to perform sentiment analysis and build topic models. Specifically, Python NLP libraries (e.g., NLTK, spaCy) are used to analyze the meaning of the text. For image and video data, computer vision techniques are used, such as OpenCV and TensorFlow, to extract features for facial recognition and object detection.

[0796] Step 5:

[0797] The server generates business process optimization proposals based on the analysis results. For example, if there are many complaints about a particular product, it uses a rule engine to identify the cause and propose improvement measures.

[0798] Step 6:

[0799] The user checks the business process optimization proposals from the server through their terminal. Specifically, an interface is provided for reviewing the proposals and approving or adjusting them as necessary. If the user is satisfied with the proposals, they click the approval button to put them into action.

[0800] Step 7:

[0801] The server automatically classifies and manages documents. For example, it automatically analyzes uploaded documents such as contracts and customer feedback documents, attaches relevant tags, and moves them to the appropriate folders, allowing users to search more efficiently.

[0802] Step 8:

[0803] Users can search for and access the documents they need. Specifically, they can use keyword search and tag filter functions to easily find the documents they are looking for. They can then select the documents they need from the search results, access them, and obtain the information they need.

[0804] Step 9:

[0805] The terminal receives inquiries from customers, analyzes the content using natural language understanding (NLU) technology, and sends the inquiry content to the server.

[0806] Step 10:

[0807] The server generates an appropriate response based on the inquiry. Specifically, it searches for similar cases using past inquiry data and a knowledge base, and generates an automatic response. The generated response is automatically sent to the customer.

[0808] Step 11:

[0809] The user checks the automated response generated by the server, makes any necessary corrections, and then sends the final reply. Specifically, the user checks the response text on the terminal interface, corrects it, and then clicks the send button to provide accurate information to the customer.

[0810] By executing each processing step sequentially in this way, companies can achieve efficient data management and analysis, optimize automated business processes, and respond quickly and appropriately to customers.

[0811] Example 1

[0812] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0813] Modern companies are required to efficiently collect and analyze information from diverse data sources and quickly and accurately optimize their business processes. However, conventional systems require a great deal of time and effort to integrate and analyze data in different formats, and there are a lack of efficient ways to improve business processes. Furthermore, automation of document management and customer support is insufficient, resulting in increased effort. The purpose of this invention is to solve these problems.

[0814] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0815] In this invention, the server includes means for collecting text, audio, image, and video data from various data sources, means for preprocessing the collected data, transcribing it, standardizing its resolution, normalizing it, and converting it into a standard format, means for analyzing the preprocessed data and proposing business process optimization using natural language processing and computer vision technologies, means for performing automated document management, converting documents into text using OCR technology, and classifying them, and means for automating customer support using natural language understanding technology. This makes it possible to efficiently integrate and analyze data in different formats, quickly optimize business processes, and automate document management and customer support.

[0816] A "data source" is a source of information used by a system to gather information.

[0817] "Preprocessing" refers to a series of processes to format collected data so that it is easier to analyze.

[0818] "Transcription" is the process of converting audio data into text data.

[0819] "Resolution unification" is a process of unifying the resolution of image data to a certain standard.

[0820] "Normalization" is the process of removing unnecessary characters and spaces from text data and formatting it into a consistent format.

[0821] "Natural language processing" is a technology that allows computers to understand and analyze human language and extract meaning.

[0822] "Computer vision technology" is a technology that analyzes image and video data to extract necessary information.

[0823] "Business process optimization" is the process of proposing improvements and streamlining of business processes based on the results of analysis.

[0824] "Automated document management" is a system that automates processes such as document classification, storage, and retrieval.

[0825] "OCR technology" is an abbreviation for Optical Character Recognition, and is a technology that extracts character information from image data.

[0826] "Natural language understanding technology" is a technology that allows computers to understand human language and generate appropriate responses and actions.

[0827] "Customer response automation" is the process by which a system automatically generates and responds to customer inquiries.

[0828] This invention is an AI-driven automation system that collects text, audio, image, and video data from multiple data sources, and preprocesses and analyzes this data to optimize corporate business processes.

[0829] System Configuration

[0830] This system is mainly composed of three components: a server, a terminal, and a user. The server is primarily responsible for data collection, preprocessing, analysis, business process optimization, document management, and customer support. The terminal functions as the user interface, inputting and outputting data. The user operates the system, uploading data, and confirming and approving proposal results.

[0831] System action

[0832] Data collection

[0833] The server periodically collects text, audio, image, and video data from data sources such as email servers, CRM systems, and customer support recording databases. The specific software used includes data collection agents and API interfaces. Users use devices to upload data, such as customer feedback and internal reports, to the system.

[0834] Data Preprocessing

[0835] The server preprocesses the collected data. For audio data, it transcribes it using speech recognition software (e.g., speech recognition API). For image data, it uses image processing software (e.g., OpenCV) to standardize the resolution. For text data, it normalizes it using a natural language processing toolkit (e.g., NLTK) to remove unnecessary characters and spaces.

[0836] Data analysis

[0837] The preprocessed data is then analyzed by the server. Natural language processing tools (e.g., spaCy) are used to perform semantic analysis on text data, and sentiment analysis and topic models are built. Image and video data are analyzed using computer vision techniques (e.g., TensorFlow) to extract the necessary information. This allows issues within the business process to be identified.

[0838] Business process optimization

[0839] Based on the analysis results, the server generates business process optimization proposals. If many customers have reported similar problems, it will identify the cause and propose specific improvement measures (for example, updating FAQs or changing processes). Users can check the proposals on their devices and approve or modify them as necessary.

[0840] Document Management

[0841] The server uses an automated document management system (e.g., a document management platform) to efficiently manage internal and external documents. Uploaded documents are converted into text using OCR technology (e.g., optical character recognition tools), and automatically classified and stored. Users can easily access the documents they need using tag and keyword searches.

[0842] Automating customer interactions

[0843] The terminal receives inquiries from customers and immediately sends the details to the server. The server uses natural language understanding technology (e.g., language understanding models) to analyze the inquiry and automatically generate an appropriate response. Responses are sent automatically, but users can review and modify complex inquiries.

[0844] Specific examples

[0845] For example, if a company receives similar complaints from multiple customers at its support desk, the system works as follows:

[0846] 1. The server periodically collects new complaint emails from the support mailbox.

[0847] 2. The server preprocesses the email data and normalizes the text data.

[0848] 3. The server uses natural language processing to analyze the content of the complaint and extract key topics and frequently occurring keywords.

[0849] 4. Based on the analysis results, the server identifies where problems are concentrated in specific products or services and generates improvement proposals.

[0850] 5. The user checks the proposed improvements on the device and approves them if necessary.

[0851] 6. Complaint data will be automatically categorized and saved in relevant folders for future reference.

[0852] 7. When a new inquiry comes in from a customer, the server generates an automatic response and replies to the customer quickly.

[0853] Prompt Sentence Examples

[0854] Example of an input prompt for a generative AI model:

[0855] "Generate optimization proposals from the following workflow:

[0856] 1. Receiving and processing customer complaints

[0857] 2. Internal Quality Assurance Process

[0858] 3. Product Development Cycle

[0859] Please provide a summary of your analysis and specific suggestions for improvement.

[0860] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0861] Step 1: Data collection

[0862] The server periodically collects text, voice, image, and video data from data sources such as email servers, CRM systems, and customer support recording databases. The server retrieves this data through an API interface or by using a data collection agent and stores it in data storage. Users use their devices to upload data such as customer feedback and internal reports and send it to the server. The input of this step is raw data such as emails, voice recordings, and images, and the output is the collected, unprocessed data.

[0863] Step 2: Data Preprocessing

[0864] The server preprocesses the collected raw data. For audio data, it converts it into text using speech recognition software (e.g., a speech recognition API). For image data, it uses image processing software (e.g., OpenCV) to standardize the resolution and perform noise reduction if necessary. Furthermore, it normalizes the text data using a natural language processing toolkit (e.g., NLTK) to remove unnecessary characters and spaces. The input of this step is the raw data, and the output is the preprocessed data.

[0865] Step 3: Data analysis

[0866] The server analyzes the preprocessed data. Text data is semantically analyzed using natural language processing tools (e.g., spaCy) to perform sentiment analysis and build topic models. Image data and video data are analyzed using computer vision techniques (e.g., TensorFlow) to extract the necessary information. The analysis results include the data's topics, sentiment scores, and the detection of specific events and objects. The input to this step is the preprocessed data, and the output is the analysis results.

[0867] Step 4: Optimize business processes

[0868] The server generates optimization proposals for business processes based on the results of data analysis. For example, if many complaints with low sentiment scores are concentrated around a specific product, the server identifies the cause and proposes countermeasures (e.g., FAQ updates, product modifications). The user can review these proposals on their device and approve or modify them as necessary. The input to this step is the analysis results, and the output is optimization proposals.

[0869] Step 5: Document Management

[0870] The server uses an automated document management system (e.g., a document management platform) to efficiently manage internal and external documents. It converts uploaded documents into text using OCR technology (e.g., optical character recognition tools), classifies them based on categories, and stores them in relevant folders. Users can access the documents they need using a search interface. The input of this step is document data, and the output is classified and stored documents.

[0871] Step 6: Automate customer interactions

[0872] The terminal receives customer inquiries in real time and immediately sends the content to the server. The server uses natural language understanding technology (e.g., a language understanding model) to analyze the inquiry and generate an appropriate automated response. For simple inquiries, an automated response is sent to the customer immediately, but for complex inquiries, the user can review and modify the response. The input to this step is the text of the customer inquiry, and the output is the generated response.

[0873] (Application example 1)

[0874] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0875] In manufacturing processes within factories, the collection and analysis of a wide variety of data (text, audio, images, video) is not being carried out efficiently, making it difficult to optimize business processes and not fully automating document management and customer support. This makes it difficult to improve business efficiency and customer satisfaction. To solve this problem, an automated system that uses integrated data management and advanced analysis technology is required.

[0876] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0877] In this invention, the server includes a means for collecting text, audio, image, and video data, a means for preprocessing the collected data and converting it into a standard format, and a means for analyzing the preprocessed data and proposing business process optimization. This enables data analysis and optimization proposals to be made using a clustering algorithm in a manufacturing environment. The server also includes a means for converting audio data into text data and analyzing it, and a means for collecting and preprocessing data from various sensors, cameras, and audio recorders. Furthermore, by incorporating a means for analyzing the sentiment of audio data using natural language processing technology and improving the efficiency of customer service, it is possible to improve not only business efficiency but also customer satisfaction.

[0878] "Text data" refers to character string information written in a natural language, and includes formats such as documents, emails, and reports.

[0879] "Voice data" refers to digitally recorded human speech or acoustic signals, and is used for speech recognition and emotion analysis.

[0880] "Image data" is visual information stored in digital form, including photographs, graphics, and diagrams.

[0881] "Moving image data" refers to data that digitally records visual information that changes over time, including video clips and video recordings.

[0882] "Preprocessing" refers to a series of operations to convert collected data into a format suitable for analysis, including format standardization and noise removal.

[0883] A "clustering algorithm" is a machine learning technique for classifying data into multiple clusters, based on patterns and similarities in the data.

[0884] "Natural language processing" is a technology that enables computers to understand and analyze human language, and is used for semantic and sentiment analysis of text.

[0885] "Sentiment analysis" is a technology that detects and classifies emotions from text and voice data, and is useful for automating customer responses.

[0886] A "sensor" is a device that detects the physical state of an environment or object and is used to collect data.

[0887] A "camera" is a device that captures visual information and stores it as digital data, and is used to collect image data and video data.

[0888] "Audio recording device" means a device for recording audio in digital form and is used to collect speech and acoustic data.

[0889] This invention is an AI-driven automation system that aims to improve the efficiency of work processes in a factory. In an embodiment of the invention, the system is composed of three entities: a server, a terminal, and a user.

[0890] System Configuration

[0891] The server plays the primary role of collecting data, preprocessing, analyzing, optimizing, managing, and responding to customers. The terminal is a device that inputs and outputs data and provides a user interface, and is also used for robots installed on production lines in factories. The user operates the system, uploads data, and checks and approves proposed results.

[0892] Program processing overview

[0893] Data collection

[0894] The server periodically collects text data, audio data, image data, and video data from various sensors, cameras, and audio recorders in the factory. For example, the server acquires this data from the production line's recording devices and quality control systems. Users use their terminals to upload data such as production daily reports and error logs.

[0895] Data Preprocessing

[0896] The server preprocesses the collected data. For example, it converts audio data to text data and standardizes the resolution of image data. Text data is normalized and unnecessary characters and spaces are removed. This ensures that the data is in a consistent format and suitable for analysis.

[0897] Data analysis

[0898] The preprocessed data is then analyzed by the server. Natural language processing is used to perform semantic and sentiment analysis of the text data and build a topic model. A clustering algorithm is also used to classify the manufacturing data into clusters. Image and video data is analyzed using computer vision technology to extract the necessary information. This allows issues in the manufacturing process within the factory to be identified.

[0899] Business process optimization

[0900] Based on the analysis results, the server generates business process optimization proposals. For example, if multiple manufacturing errors occur for the same reason, the server will identify the cause and propose improvement measures. Users can check the proposals on their devices and approve or modify them as necessary.

[0901] Document Management

[0902] The server uses an automatic document management system to efficiently manage documents inside and outside the factory. Uploaded documents are automatically categorized and placed in relevant folders. Users can easily access the documents they need using tag or keyword searches.

[0903] Automating customer interactions

[0904] The terminal receives inquiries from customers and immediately sends them to the server, which uses natural language understanding technology to analyze the inquiries and generate appropriate responses. Responses are automatically sent, and users can review and modify them as needed.

[0905] Specific examples

[0906] As a concrete example, when quality control is performed on a manufacturing line in a factory, the system operates as follows.

[0907] 1. The server periodically collects new data from the production line's sensor data and quality control records.

[0908] 2. The server preprocesses these data and converts them into a standard format.

[0909] 3. The server analyzes the data using natural language processing and clustering algorithms to identify key manufacturing challenges and frequent problems.

[0910] 4. Based on the analysis results, the server identifies where problems are concentrated in specific manufacturing processes and generates improvement proposals.

[0911] 5. The user checks the proposed improvements on the device and approves them if necessary.

[0912] 6. Manufacturing data is automatically categorized and saved in relevant folders for future reference.

[0913] Prompt Sentence Examples

[0914] "Collect new complaint emails from the support mailbox, analyze the content of the complaint using natural language processing, and generate a report of the results."

[0915] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0916] Step 1:

[0917] Data collection

[0918] The server periodically collects text data, voice data, image data, and video data from various sensors, cameras, and audio recorders in the factory. For example, the server receives inputs from the production line monitoring system and quality control system to acquire these data, and outputs the collected data to be stored in the centralized management system.

[0919] Step 2:

[0920] Data Preprocessing

[0921] The server preprocesses the collected data. For example, it converts audio data into text data and standardizes the resolution of image data. Specifically, it uses voice recognition software to convert audio data into text and image processing algorithms to standardize the resolution of images. This preprocessing ensures that the data is in a consistent format, providing appropriate input data for analysis. The output is standardized text, audio, image, and video data.

[0922] Step 3:

[0923] Data analysis

[0924] The preprocessed data is then analyzed by the server, which uses natural language processing to perform semantic and sentiment analysis of the text data. For example, a generative AI model is used to detect sentiment from the text data and build a topic model. A clustering algorithm is also used to classify the manufacturing data into clusters. Image and video data is analyzed using computer vision techniques to extract information for anomaly detection and quality assessment on the production line. The output of this step is a detailed report based on the analysis results.

[0925] Step 4:

[0926] Business process optimization

[0927] Based on the analysis results, the server generates optimization proposals for business processes. The server identifies patterns of manufacturing errors and frequent problems and proposes specific improvement measures. It also uses a customized generative AI model to generate optimization proposals. These optimization proposals include specific instructions and action plans for improving the manufacturing process, thereby enabling efficient business operations. The output is a proposal for improving the business process.

[0928] Step 5:

[0929] Document Management

[0930] The server uses an automatic document management system to efficiently manage documents inside and outside the factory. Uploaded documents are automatically categorized and placed in relevant folders. They are also designed to be easily accessible using tag and keyword searches, allowing users to quickly access the documents they need. This improves the efficiency of document management. The output is categorized and organized document data.

[0931] Step 6:

[0932] Automating customer interactions

[0933] The terminal receives an inquiry from a customer and immediately sends the content to the server. The server uses natural language understanding technology to analyze the content of the inquiry and generate an appropriate response. The generated response is sent automatically, but the user can confirm and modify it if necessary. A prompt sentence (e.g., "Please collect new complaint emails from the support mailbox, analyze the content of the complaint using natural language processing, and generate a report of the results") may also be used to generate an initial response. The output of this step is a prompt and appropriate response to improve customer satisfaction.

[0934] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0935] This invention is an AI-driven automation system aimed at improving the efficiency of corporate operations, and in particular, by combining it with an emotion engine that recognizes user emotions, it achieves more advanced optimization of business processes and automation of customer support. In the embodiments of the present invention, a method is described for integrating and analyzing text, audio, image, and video data to automate optimization of business processes, document management, and customer support that take user emotions into consideration.

[0936] System Configuration

[0937] The system mainly consists of three components: a server, a terminal, and a user. The server is primarily responsible for data collection, preprocessing, analysis, optimization, management, customer service, and emotion recognition. The terminal is a device that inputs and outputs data and provides a user interface. The user operates the system, uploads data, and reviews and approves proposal results.

[0938] Program processing overview

[0939] Data collection

[0940] The server collects data from various sources, including email servers, CRM systems, and customer support recording databases, periodically retrieving text, audio, image, and video data. Users use their devices to upload data, such as customer feedback and internal reports, to the system.

[0941] Data Preprocessing

[0942] The server preprocesses the collected data and converts it into a standard format. For example, audio data is transcribed, image data is standardized, and text data is normalized to remove unnecessary characters and whitespace. This ensures the data is in a consistent format and suitable for analysis.

[0943] Data Analysis and Emotion Recognition

[0944] The preprocessed data is then analyzed by the server. Natural language processing is used to perform semantic analysis of the text data, and sentiment analysis is also performed simultaneously. For example, positive, negative, and neutral emotions are extracted from the text data. Voice data is also analyzed using a sentiment extraction engine to recognize the customer's emotions. Image and video data are also analyzed using computer vision technology to identify emotions from the user's facial expressions.

[0945] Business process optimization

[0946] Based on the analysis results and the recognized emotions, the server generates business process optimization proposals. For example, if many customers have negative emotions about a particular product, the server will identify the cause and propose improvements. The user can review the proposals on their device and approve or modify them as necessary.

[0947] Document Management

[0948] The server uses an automatic document management system to efficiently manage internal and external documents. Uploaded documents are automatically categorized and placed in relevant folders. Users can easily access the documents they need using tag or keyword searches.

[0949] Automating customer interactions

[0950] The terminal receives inquiries from customers and immediately sends them to the server. The server uses natural language understanding technology to analyze the inquiries and generate appropriate responses. Responses can also be tailored based on the user's sentiment. Responses are automatically sent, and users can review and modify them as needed.

[0951] Specific examples

[0952] For example, if a company receives similar complaints from multiple customers at its support desk, the system works as follows:

[0953] 1. The server periodically collects new complaint emails from the support mailbox.

[0954] 2. The server preprocesses the email data and normalizes the text data.

[0955] 3. The server uses natural language processing to analyze the content of the complaint and extract key topics and frequently occurring keywords.

[0956] 4. At the same time, the server uses an emotion engine to recognize customer emotions from the text data. For example, if many emails express negative emotions, this is detected.

[0957] 5. Based on the analysis results and sentiment data, the server identifies problems concentrated in specific products or services and generates improvement proposals. For example, if there is a lot of negative feedback stating that "there are problems with the quality of a specific product," it will propose improvement measures for that product.

[0958] 6. The user checks the proposed improvements on the terminal and approves them if necessary.

[0959] 7. Complaint data will be automatically categorized and saved in relevant folders for future reference.

[0960] 8. When a new inquiry comes in from a customer, the server generates an automatic response and quickly replies to the customer with a message that reflects the user's feelings.

[0961] By implementing this invention, enterprises can significantly improve their business efficiency and provide high-quality customer service that takes into consideration users' emotions.

[0962] The processing flow will be explained below.

[0963] Step 1:

[0964] The server collects data from various sources: email data from the mail server, customer interaction data from the CRM system, and voice data from the recording database. The server periodically checks these data sources and collects any new data.

[0965] Step 2:

[0966] Users upload data to the system using their devices. Specifically, they drag and drop customer feedback audio files and image data into a dedicated upload interface and send them to the server. Users do this as part of their daily work.

[0967] Step 3:

[0968] The server preprocesses the collected data. For example, it transcribes audio data and standardizes the resolution of image data. It normalizes text data and removes unnecessary characters and whitespace. This preprocessing ensures that the data is in a consistent format and suitable for analysis.

[0969] Step 4:

[0970] The server analyzes the preprocessed data. Specifically, it uses natural language processing (NLP) to perform semantic analysis of the text data. It uses Python NLP libraries (e.g., spaCy, NLTK) to analyze the structure of the text and build sentiment analysis and topic models. It also uses speech recognition and emotion recognition engines to extract emotions from audio data.

[0971] Step 5:

[0972] The server analyzes image and video data. It uses computer vision technology (e.g., OpenCV, TensorFlow) to analyze the image data and identify emotions from the user's facial expressions. Similarly, it extracts emotions from video data based on specific facial expressions and movements.

[0973] Step 6:

[0974] The server uses an emotion recognition engine to extract the user's emotion from the analyzed data. For text data, it calculates an emotion score such as positive, negative, or neutral, and generates similar emotion scores for audio and image data.

[0975] Step 7:

[0976] The server generates business process optimization proposals based on the analysis results and sentiment data. For example, if many customers have negative feelings about a particular product, the server identifies the cause and proposes improvement measures. The proposals are generated in the form of specific action plans and changes.

[0977] Step 8:

[0978] The user checks the business process optimization proposals from the server through their terminal. For example, they review the proposals displayed on the terminal and approve or adjust them as necessary. If the user is satisfied with the proposals, they click the approval button to put them into action.

[0979] Step 9:

[0980] The server automatically classifies and manages documents. For example, it automatically analyzes uploaded documents such as contracts and customer feedback documents, attaches relevant tags, and moves them to the appropriate folders, improving document search efficiency.

[0981] Step 10:

[0982] Users search for and access the documents they need. Specifically, they use the device's search function to enter keywords or tags to quickly identify the desired document. They then select the relevant document from the search results and access it to obtain the information they need.

[0983] Step 11:

[0984] The terminal receives inquiries from customers, analyzes the content using natural language understanding (NLU) technology, and sends the inquiry to a server. For example, this is done by chatbots and email reception interfaces.

[0985] Step 12:

[0986] The server generates an appropriate response based on the inquiry. Specifically, it searches for similar cases using past inquiry data and a knowledge base, and generates an automatic response. The generated response is quickly sent to the customer.

[0987] Step 13:

[0988] The user checks the automated response generated by the server, makes any necessary corrections, and then sends the final reply. For example, by checking the response text on the terminal interface, correcting it, and then clicking the send button, the user can provide accurate information to the customer.

[0989] Example 2

[0990] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0991] Modern business activities require the efficient management and analysis of massive amounts of data, as well as the optimization of business processes and the automation of customer responses. In particular, an approach that handles text, voice, image, and video data in an integrated manner and takes user emotions into account is essential, but there are currently no systems that can effectively achieve this. Furthermore, managing and analyzing data manually is time-consuming, costly, and prone to human error. Therefore, there is a need for a comprehensive system that automates everything from data collection and analysis to optimization proposals, document management, and customer responses.

[0992] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0993] In this invention, the server includes means for collecting text, audio, image, and video data from various data sources, means for preprocessing the collected data and converting it into a standard format, and means for analyzing the preprocessed data using a natural language processing and emotion recognition engine. This enables companies to efficiently manage and analyze massive amounts of data, optimize business processes, and automate customer responses.

[0994] "Various data sources" refers to multiple information sources, including email servers, customer relationship management systems, customer support recording databases, etc.

[0995] "Text data" refers to data expressed in text format, such as emails, reports, and chat history.

[0996] "Audio data" refers to data expressed in the form of voice, such as audio files and recorded data.

[0997] "Image data" refers to data expressed in a visual format, such as photographs, screenshots, and illustrations.

[0998] "Video data" refers to data that is presented in a moving visual format, such as a video clip or recorded data.

[0999] "Collect" refers to the act of obtaining data and putting it into a system.

[1000] "Preprocessing" refers to the initial processing of collected data to convert it into a form that is easier to analyze. Examples include transcribing audio data and normalizing text data.

[1001] "Standard format" refers to a standard for standardizing data into a consistent format.

[1002] "Natural language processing" refers to the technology that enables computers to understand and analyze natural language written and spoken by humans.

[1003] An "emotion recognition engine" refers to software or algorithms that identify and analyze emotions contained within data.

[1004] "Analyzing" refers to the act of analyzing data to understand its content and patterns and extract information.

[1005] "Business process optimization" refers to the act of planning and implementing proposals and measures to improve the efficiency of corporate activities.

[1006] "Automated document management" refers to the automatic performance of management tasks such as document classification, storage, and retrieval by a computer system.

[1007] "Customer response automation" refers to a system automatically generating and sending appropriate responses to customer inquiries.

[1008] This invention is an AI-driven automation system aimed at improving business efficiency, and in particular, by combining an emotion engine that recognizes user emotions, it optimizes business processes and automates customer support. This system is mainly composed of three components: a server, a terminal, and a user. Specific embodiments of this system are described below.

[1009] System Configuration

[1010] The system consists of three components: a server, a terminal, and a user. The server is primarily responsible for data collection, preprocessing, analysis, optimization, management, customer support, and emotion recognition. The terminal is a device that inputs and outputs data and provides a user interface. The user operates the system, uploads data, and confirms and approves proposal results. Specific implementation examples of each component are shown below.

[1011] server

[1012] The server periodically collects text, audio, image, and video data from various data sources (e.g., email servers, CRM systems, customer support recording databases, etc.). The collected data is ingested through an ETL (Extract, Transform, Load) process. The server then preprocesses the collected data. Audio data is transcribed using a speech-to-text engine (e.g., Google Cloud Speech-to-Text), image data is standardized in resolution, and video data is broken down into frames. Text data is normalized, and unnecessary characters and whitespace are removed. Regular expressions and natural language processing tools (e.g., NLTK) are used in this process.

[1013] The preprocessed data is then analyzed using a natural language processing (NLP) engine (e.g., BERT or GPT model) to perform semantic analysis of the text data. A sentiment analysis engine (e.g., Sentiment140) is used to extract positive, negative, or neutral sentiment. Audio data is also analyzed using a sentiment extraction engine. Finally, image and video data are analyzed using computer vision techniques (e.g., OpenCV or deep learning models) to identify emotions from the user's facial expressions.

[1014] Based on the analysis results, the server generates business process optimization proposals. For example, if many customers have negative feelings about a particular product, it will suggest improvements for that product. It also uses an automated document management system to efficiently manage uploaded documents. Documents are categorized into appropriate folders, and necessary documents can be quickly accessed using tag or keyword searches.

[1015] Terminal

[1016] The terminal is a device that allows users to input and output data and provides a user interface. Users upload data through the terminal and review and approve proposals and analysis results. When users upload data, the terminal checks the data format and displays a warning if the format is inappropriate. The terminal also receives inquiries from customers and immediately sends the details to the server.

[1017] User

[1018] Users operate the system to upload data and review and approve proposal results. Specifically, they use terminals to upload data such as customer feedback and internal reports to the system. They also review the proposal content and approve or modify it if necessary. When a customer inquiry is received, they can also review the generated response and make modifications as necessary.

[1019] Specific examples

[1020] For example, if a company receives similar complaints from multiple customers at its support desk, the system works as follows:

[1021] 1. The server periodically collects new complaint emails from the support mailbox.

[1022] 2. The server preprocesses the email data and normalizes the text data.

[1023] 3. The server uses natural language processing to analyze the content of the complaint and extract key topics and frequently occurring keywords.

[1024] 4. At the same time, the server uses an emotion engine to recognize customer emotions from the text data. For example, if many emails express negative emotions, this is detected.

[1025] 5. Based on the analysis results and sentiment data, the server identifies problems concentrated in specific products or services and generates improvement proposals. For example, if there is a lot of negative feedback stating that "there are problems with the quality of a specific product," it will propose improvement measures for that product.

[1026] 6. The user checks the proposed improvements on the terminal and approves them if necessary.

[1027] 7. Complaint data will be automatically categorized and saved in relevant folders for future reference.

[1028] 8. When a new inquiry comes in from a customer, the server generates an automatic response and quickly replies to the customer with a message that reflects the user's feelings.

[1029] Prompt Sentence Examples

[1030] Below are some examples of text data to analyze and specific prompts to extract sentiment:

[1031] Prompt: "Conduct a customer feedback analysis and generate a report that extracts key topics and their respective sentiments. Analyze the following feedback: 'Product X is difficult to use. Support is slow to respond.'"

[1032] By feeding this prompt into a generative AI model, the system can analyze the main topics of the feedback (e.g., usability, support response) and sentiment (negative) and generate a corresponding report.

[1033] In this way, the present invention significantly improves the business efficiency of companies, and in particular enables high-quality responses that take into account the user's emotions.

[1034] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1035] Step 1:

[1036] The server collects text, audio, image, and video data from various data sources (e.g., email servers, CRM systems, customer support recording databases, etc.) and periodically ingests new data by running an ETL (Extract, Transform, Load) process. This step uses data from various data sources as input and produces the collected data as output.

[1037] Step 2:

[1038] The server preprocesses the collected data. For example, audio data is transcribed using a speech-to-text engine, image data is standardized in resolution, and video data is broken down into frames. Text data is normalized using regular expressions and natural language processing tools. This step uses the collected raw data as input and produces preprocessed data as output.

[1039] Step 3:

[1040] The server analyzes the preprocessed data. It uses a natural language processing engine (e.g., a GPT model) to perform semantic analysis of the text data and a sentiment analysis engine to extract positive, negative, or neutral sentiment. Audio data is also analyzed using a sentiment extraction engine, and image and video data are analyzed using computer vision techniques. This step uses the preprocessed data as input and generates analysis results as output.

[1041] Step 4:

[1042] The server generates business process optimization proposals based on the analysis results. For example, if a particular product has received a lot of negative feedback, it generates improvement proposals for that product. It also includes automated processes for customer support. This step uses the analysis results as input and generates optimization proposals and automated responses as output.

[1043] Step 5:

[1044] The user uses a terminal to review the improvement proposals and the automated response. The terminal displays the proposals and provides an interface for the user to approve or modify them as needed. If the user makes modifications, the modified proposals are fed back into the system. This step uses the optimization proposals and automated response as inputs and generates the proposals that have been reviewed and modified by the user as output.

[1045] Step 6:

[1046] The server uses an automated document management system to classify and store the uploaded documents. The documents are categorized into appropriate folders and can be efficiently accessed using tag and keyword searches. This step uses the uploaded documents as input and produces classified and stored documents as output.

[1047] Step 7:

[1048] When the terminal receives a customer inquiry, it immediately sends the content to the server, which analyzes the inquiry and generates an appropriate response using natural language understanding technology. The response can also be tailored based on the user's sentiment. This step uses the customer inquiry information as input and provides the generated response as output.

[1049] (Application example 2)

[1050] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1051] In conventional corporate business processes, it was difficult to analyze user emotions in real time and provide optimal responses. Furthermore, when it came to displaying advertisements, static ads were often displayed without considering user emotions, making it impossible to deliver advertisements effectively. This resulted in problems such as lower customer satisfaction and reduced business efficiency.

[1052] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1053] In this invention, the server includes means for collecting text, audio, image, and video data, means for preprocessing the collected data and converting it into a standard format, means for analyzing the preprocessed data and proposing business process optimization, means for automated document management, means for automating customer support, means for tracking user gaze and analyzing emotions in real time, and means for displaying optimal advertisements based on the analyzed emotions. This allows business processes and advertisement display to be optimized based on user emotions, enabling highly efficient and high-quality customer support.

[1054] "Text data" refers to data that contains sentences or character information, and includes emails, chat messages, documents, and the like.

[1055] "Voice data" means data obtained by recording a human voice or other sounds, including voice messages, telephone recordings, and customer support audio recordings.

[1056] "Image data" refers to visual information recorded as still images, including photographs, screenshots, illustrations, etc.

[1057] "Movie data" refers to dynamic visual information consisting of a series of image frames, including video clips and video recordings.

[1058] "Collection methods" are the technical means used to obtain text, audio, image, and video data from a variety of sources, including database access and use of APIs.

[1059] "Preprocessing means" refers to technology that converts collected data into a format that is easier to analyze, such as transcribing audio data or standardizing image resolution.

[1060] A "standard format" is a format in which data of different formats is converted into a consistent format that can be analyzed, making data analysis easier.

[1061] "Means of analysis" refers to technologies that use preprocessed data to extract information and optimize or propose business processes, including machine learning and natural language processing.

[1062] "Means for proposing optimization" refers to techniques that propose methods and means for improving the efficiency of business processes based on the analysis results, and includes formulating plans and presenting improvement measures.

[1063] "Means for document management" refers to technology for efficiently managing documents, including automatic classification and tagging of documents, and search functions.

[1064] "Means for automating customer support" refers to technology that automatically responds to inquiries and requests from customers, and includes chatbots and automatic reply systems.

[1065] "Gaze tracking means" refers to technology that detects the direction of a user's gaze and monitors it in real time, and includes gaze tracking cameras and sensors.

[1066] "Means for analyzing emotions in real time" refers to technology that detects the user's emotions at that time from their facial expressions, voice, and text, and includes facial expression recognition engines and emotion analysis algorithms.

[1067] The "means for displaying advertisements" refers to technology that displays appropriate advertisements to users based on the analyzed emotions, and includes displays and marketing engines.

[1068] This invention is an AI-driven automation system aimed at improving business efficiency and optimizing customer service. The system includes means for collecting text, voice, image, and video data, means for preprocessing the collected data and converting it into a standard format, means for analyzing the preprocessed data and proposing business process optimization, means for automated document management, and means for automating customer service. It also includes means for tracking a user's gaze and analyzing their emotions in real time, and means for displaying optimal advertisements based on the analyzed emotions.

[1069] System Configuration

[1070] This system is mainly composed of three components: a server, a terminal, and a user. The server is primarily responsible for data collection, preprocessing, analysis, optimization, emotion recognition, and display of advertisements. The terminal is a device that inputs and outputs data and provides a user interface. The user operates the system, uploads data, and reviews and approves the proposed results.

[1071] Data collection

[1072] The server collects text, audio, image, and video data from various data sources, such as email servers, CRM systems, and customer support recording databases, and users use their devices to upload data to the system, including customer feedback and internal reports.

[1073] Data Preprocessing

[1074] The server preprocesses the collected data and converts it into a standard format. For example, audio data is transcribed, image data is standardized, and text data is normalized to remove unnecessary characters and whitespace. This ensures the data is in a consistent format and suitable for analysis.

[1075] Data Analysis and Emotion Recognition

[1076] The preprocessed data is then analyzed by the server. Natural language processing is used to perform semantic analysis of the text data, and sentiment analysis is also performed simultaneously. For example, positive, negative, and neutral emotions are extracted from the text data. Voice data is also analyzed using a sentiment extraction engine to recognize customer emotions. Image and video data are also analyzed using computer vision technology to identify emotions from the user's facial expressions.

[1077] Business process optimization

[1078] Based on the analysis results and the recognized emotions, the server generates business process optimization proposals. For example, if many customers have negative emotions about a particular product, the server will identify the cause and propose improvements. The user can review the proposals on their device and approve or modify them as necessary.

[1079] Document Management

[1080] The server efficiently manages internal and external documents using an automatic document management system. Uploaded documents are automatically categorized and placed in relevant folders. Users can easily access the documents they need using tag or keyword searches.

[1081] Automating customer interactions

[1082] The terminal receives inquiries from customers and immediately sends them to the server. The server uses natural language understanding technology to analyze the inquiries and generate appropriate responses. It can also adjust the responses based on the user's sentiment. Responses are automatically sent, and users can review and modify them as needed.

[1083] Eye Tracking and Emotion Analysis

[1084] When a user wears smart glasses, the eye tracking camera detects the direction of their gaze and analyzes their facial expressions in real time. The server collects this data and performs emotion analysis. Based on the analyzed emotion, the server selects the most appropriate advertisement and displays it in the user's field of view.

[1085] Specific examples

[1086] When a user wears smart glasses and browses web content, Serve analyzes their gaze and facial expressions in real time. For example, if a user shows interest in a particular product page, Serve can detect positive emotions and display relevant ads based on those emotions. This maximizes the effectiveness of advertising and improves the user experience.

[1087] An example of a prompt is, "Design an application that recognizes emotions in real time from the gaze and facial expressions captured by the camera when the user is wearing smart glasses, and displays the most appropriate advertisement based on that emotion. Specifically, implement a function that uses emotion recognition technology to detect the main emotion and selects and displays an appropriate advertisement image. In addition, the advertisement image will be selected from a list that has been pre-classified into several emotional categories."

[1088] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1089] Step 1: Data collection

[1090] The server periodically retrieves text, audio, image, and video data from mail servers, CRM systems, and customer support recording databases. Users use their devices to upload data, such as customer feedback and internal reports, to the system. The input is various data from external data sources, and the output is the collected raw data. Specifically, the server retrieves data using API calls and database queries.

[1091] Step 2: Preprocessing the data

[1092] The server preprocesses the collected data and converts it into a standard format. Specifically, it transcribes audio data, standardizes the resolution of image data, and normalizes text data to remove unnecessary characters and spaces. The input is the raw data collected, and the output is the preprocessed data. The server performs these conversions using a speech recognition engine and image processing algorithms.

[1093] Step 3: Data analysis and emotion recognition

[1094] The server analyzes the preprocessed data and uses natural language processing to perform semantic analysis of the text data, while simultaneously performing sentiment analysis. For example, positive, negative, and neutral emotions are extracted from the text data. Voice data is analyzed using an emotion extraction engine to recognize customer emotions. Image data and video data are analyzed using computer vision technology. The input is the preprocessed data, and the output is the analysis results and emotional data. Specifically, the server performs these analyses using a natural language processing library and a facial recognition engine.

[1095] Step 4: Optimize business processes

[1096] The server generates optimization proposals for business processes based on the analysis results and the recognized emotions. For example, if many customers have negative emotions about a particular product, the cause can be identified and improvement measures proposed. The input is the analysis results and emotion data, and the output is optimization proposals for business processes. Specifically, the server generates optimization proposals using machine learning algorithms.

[1097] Step 5: Document Management

[1098] The server uses an automated document management system to efficiently manage internal and external documents. Uploaded documents are automatically categorized and placed in relevant folders. Users can access the documents they need using tag or keyword searches. The input is the uploaded document data, and the output is organized and categorized documents. Specifically, the server uses document management software.

[1099] Step 6: Automate customer interactions

[1100] The terminal receives an inquiry from a customer and immediately sends the content to the server. The server uses natural language understanding technology to analyze the inquiry and generate an appropriate response. The input is the customer's inquiry, and the output is the generated response message. Specifically, the server uses a natural language understanding engine.

[1101] Step 7: Gaze Tracking and Emotion Analysis

[1102] When a user wears smart glasses, an eye-tracking camera is used to detect the direction of gaze and analyze facial expressions in real time. The server collects this data and performs emotion analysis. The input is the user's gaze and facial expression data, and the output is analyzed emotion data. Specifically, the server uses an eye-tracking camera and facial expression recognition software.

[1103] Step 8: Optimizing Ad Display

[1104] The server selects the most suitable advertisement based on the analyzed emotion and displays it in the user's field of view. The input is the analyzed emotion data, and the output is the displayed advertisement image. Specifically, the server uses an advertisement selection algorithm and a display device.

[1105] An example of a prompt is, "Design an application that recognizes emotions in real time from the gaze and facial expressions captured by the camera when the user is wearing smart glasses, and displays the most appropriate advertisement based on that emotion. Specifically, implement a function that uses emotion recognition technology to detect the main emotion and selects and displays an appropriate advertisement image. In addition, the advertisement image will be selected from a list that has been pre-classified into several emotional categories."

[1106] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1107] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1108] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1109] [Fourth embodiment]

[1110] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1111] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1112] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1113] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1114] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1115] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1116] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1117] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1118] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1119] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1120] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1121] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1122] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1123] This invention is an AI-driven automation system aimed at improving business efficiency in companies. In the embodiment of the invention, a method for integrating and analyzing text, audio, image, and video data to automate business process optimization, document management, and customer support is described.

[1124] System Configuration

[1125] The system mainly consists of three components: a server, a terminal, and a user. The server is primarily responsible for data collection, preprocessing, analysis, optimization, management, and customer support. The terminal is a device that inputs and outputs data and provides a user interface. The user operates the system, uploads data, and reviews and approves proposal results.

[1126] Program processing overview

[1127] Data collection

[1128] The server periodically collects text, audio, image, and video data from various data sources, such as email servers, CRM systems, and customer support recording databases. Users use devices to upload data such as customer feedback and internal reports.

[1129] Data Preprocessing

[1130] The server preprocesses the collected data. For example, it transcribes audio data and standardizes the resolution of image data. It normalizes text data and removes unnecessary characters and whitespace. This ensures the data is in a consistent format and suitable for analysis.

[1131] Data analysis

[1132] The preprocessed data is then analyzed by the server. Natural language processing is used to perform semantic analysis of the text data, and sentiment analysis and topic models are built. Image and video data are also analyzed using computer vision technology to extract the necessary information. This allows issues within the business process to be identified.

[1133] Business process optimization

[1134] Based on the analysis results, the server generates business process optimization proposals. For example, if many customers report the same problem, it will identify the cause and propose improvements. Users can review the proposals on their devices and approve or modify them as necessary.

[1135] Document Management

[1136] The server uses an automatic document management system to efficiently manage internal and external documents. Uploaded documents are automatically categorized and placed in relevant folders. Users can easily access the documents they need using tag or keyword searches.

[1137] Automating customer interactions

[1138] The terminal receives inquiries from customers and immediately sends them to the server, which uses natural language understanding technology to analyze the inquiries and generate appropriate responses. Responses are automatically sent, and users can review and modify them as needed.

[1139] Specific examples

[1140] For example, if a company receives similar complaints from multiple customers at its support desk, the system works as follows:

[1141] 1. The server periodically collects new complaint emails from the support mailbox.

[1142] 2. The server preprocesses the email data and normalizes the text data.

[1143] 3. The server uses natural language processing to analyze the content of the complaint and extract key topics and frequently occurring keywords.

[1144] 4. Based on the analysis results, the server identifies where problems are concentrated in specific products or services and generates improvement proposals.

[1145] 5. The user checks the proposed improvements on the device and approves them if necessary.

[1146] 6. Complaint data will be automatically categorized and saved in relevant folders for future reference.

[1147] 7. When a new inquiry comes in from a customer, the server generates an automatic response and replies to the customer quickly.

[1148] By implementing this invention, companies can significantly improve their business efficiency and increase customer satisfaction.

[1149] The processing flow will be explained below.

[1150] Step 1:

[1151] The server collects data from various sources: email data from the mail server, customer interaction data downloaded from the CRM system, and audio data from the recording database.

[1152] Step 2:

[1153] Users upload data to the system using their devices. Specifically, they drag and drop customer feedback audio files and image data onto a dedicated upload interface to send them to the server.

[1154] Step 3:

[1155] The server preprocesses the collected data in bulk. For example, it uses a speech recognition API to convert voice data into text, an image conversion library to standardize image resolution, and normalizes the text data to remove unnecessary characters and spaces.

[1156] Step 4:

[1157] The server analyzes the preprocessed data. For text data, natural language processing (NLP) is used to perform sentiment analysis and build topic models. Specifically, Python NLP libraries (e.g., NLTK, spaCy) are used to analyze the meaning of the text. For image and video data, computer vision techniques are used, such as OpenCV and TensorFlow, to extract features for facial recognition and object detection.

[1158] Step 5:

[1159] The server generates business process optimization proposals based on the analysis results. For example, if there are many complaints about a particular product, it uses a rule engine to identify the cause and propose improvement measures.

[1160] Step 6:

[1161] The user checks the business process optimization proposals from the server through their terminal. Specifically, an interface is provided for reviewing the proposals and approving or adjusting them as necessary. If the user is satisfied with the proposals, they click the approval button to put them into action.

[1162] Step 7:

[1163] The server automatically classifies and manages documents. For example, it automatically analyzes uploaded documents such as contracts and customer feedback documents, attaches relevant tags, and moves them to the appropriate folders, allowing users to search more efficiently.

[1164] Step 8:

[1165] Users can search for and access the documents they need. Specifically, they can use keyword search and tag filter functions to easily find the documents they are looking for. They can then select the documents they need from the search results, access them, and obtain the information they need.

[1166] Step 9:

[1167] The terminal receives inquiries from customers, analyzes the content using natural language understanding (NLU) technology, and sends the inquiry content to the server.

[1168] Step 10:

[1169] The server generates an appropriate response based on the inquiry. Specifically, it searches for similar cases using past inquiry data and a knowledge base, and generates an automatic response. The generated response is automatically sent to the customer.

[1170] Step 11:

[1171] The user checks the automated response generated by the server, makes any necessary corrections, and then sends the final reply. Specifically, the user checks the response text on the terminal interface, corrects it, and then clicks the send button to provide accurate information to the customer.

[1172] By executing each processing step sequentially in this way, companies can achieve efficient data management and analysis, optimize automated business processes, and respond quickly and appropriately to customers.

[1173] Example 1

[1174] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1175] Modern companies are required to efficiently collect and analyze information from diverse data sources and quickly and accurately optimize their business processes. However, conventional systems require a great deal of time and effort to integrate and analyze data in different formats, and there are a lack of efficient ways to improve business processes. Furthermore, automation of document management and customer support is insufficient, resulting in increased effort. The purpose of this invention is to solve these problems.

[1176] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1177] In this invention, the server includes means for collecting text, audio, image, and video data from various data sources, means for preprocessing the collected data, transcribing it, standardizing its resolution, normalizing it, and converting it into a standard format, means for analyzing the preprocessed data and proposing business process optimization using natural language processing and computer vision technologies, means for performing automated document management, converting documents into text using OCR technology, and classifying them, and means for automating customer support using natural language understanding technology. This makes it possible to efficiently integrate and analyze data in different formats, quickly optimize business processes, and automate document management and customer support.

[1178] A "data source" is a source of information used by a system to gather information.

[1179] "Preprocessing" refers to a series of processes to format collected data so that it is easier to analyze.

[1180] "Transcription" is the process of converting audio data into text data.

[1181] "Resolution unification" is a process of unifying the resolution of image data to a certain standard.

[1182] "Normalization" is the process of removing unnecessary characters and spaces from text data and formatting it into a consistent format.

[1183] "Natural language processing" is a technology that allows computers to understand and analyze human language and extract meaning.

[1184] "Computer vision technology" is a technology that analyzes image and video data to extract necessary information.

[1185] "Business process optimization" is the process of proposing improvements and streamlining of business processes based on the results of analysis.

[1186] "Automated document management" is a system that automates processes such as document classification, storage, and retrieval.

[1187] "OCR technology" is an abbreviation for Optical Character Recognition, and is a technology that extracts character information from image data.

[1188] "Natural language understanding technology" is a technology that allows computers to understand human language and generate appropriate responses and actions.

[1189] "Customer response automation" is the process by which a system automatically generates and responds to customer inquiries.

[1190] This invention is an AI-driven automation system that collects text, audio, image, and video data from multiple data sources, and preprocesses and analyzes this data to optimize corporate business processes.

[1191] System Configuration

[1192] This system is mainly composed of three components: a server, a terminal, and a user. The server is primarily responsible for data collection, preprocessing, analysis, business process optimization, document management, and customer support. The terminal functions as the user interface, inputting and outputting data. The user operates the system, uploading data, and confirming and approving proposal results.

[1193] System action

[1194] Data collection

[1195] The server periodically collects text, audio, image, and video data from data sources such as email servers, CRM systems, and customer support recording databases. The specific software used includes data collection agents and API interfaces. Users use devices to upload data, such as customer feedback and internal reports, to the system.

[1196] Data Preprocessing

[1197] The server preprocesses the collected data. For audio data, it transcribes it using speech recognition software (e.g., speech recognition API). For image data, it uses image processing software (e.g., OpenCV) to standardize the resolution. For text data, it normalizes it using a natural language processing toolkit (e.g., NLTK) to remove unnecessary characters and spaces.

[1198] Data analysis

[1199] The preprocessed data is then analyzed by the server. Natural language processing tools (e.g., spaCy) are used to perform semantic analysis on text data, and sentiment analysis and topic models are built. Image and video data are analyzed using computer vision techniques (e.g., TensorFlow) to extract the necessary information. This allows issues within the business process to be identified.

[1200] Business process optimization

[1201] Based on the analysis results, the server generates business process optimization proposals. If many customers have reported similar problems, it will identify the cause and propose specific improvement measures (for example, updating FAQs or changing processes). Users can check the proposals on their devices and approve or modify them as necessary.

[1202] Document Management

[1203] The server uses an automated document management system (e.g., a document management platform) to efficiently manage internal and external documents. Uploaded documents are converted into text using OCR technology (e.g., optical character recognition tools), and automatically classified and stored. Users can easily access the documents they need using tag and keyword searches.

[1204] Automating customer interactions

[1205] The terminal receives inquiries from customers and immediately sends the details to the server. The server uses natural language understanding technology (e.g., language understanding models) to analyze the inquiry and automatically generate an appropriate response. Responses are sent automatically, but users can review and modify complex inquiries.

[1206] Specific examples

[1207] For example, if a company receives similar complaints from multiple customers at its support desk, the system works as follows:

[1208] 1. The server periodically collects new complaint emails from the support mailbox.

[1209] 2. The server preprocesses the email data and normalizes the text data.

[1210] 3. The server uses natural language processing to analyze the content of the complaint and extract key topics and frequently occurring keywords.

[1211] 4. Based on the analysis results, the server identifies where problems are concentrated in specific products or services and generates improvement proposals.

[1212] 5. The user checks the proposed improvements on the device and approves them if necessary.

[1213] 6. Complaint data will be automatically categorized and saved in relevant folders for future reference.

[1214] 7. When a new inquiry comes in from a customer, the server generates an automatic response and replies to the customer quickly.

[1215] Prompt Sentence Examples

[1216] Example of an input prompt for a generative AI model:

[1217] "Generate optimization proposals from the following workflow:

[1218] 1. Receiving and processing customer complaints

[1219] 2. Internal Quality Assurance Process

[1220] 3. Product Development Cycle

[1221] Please provide a summary of your analysis and specific suggestions for improvement.

[1222] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1223] Step 1: Data collection

[1224] The server periodically collects text, voice, image, and video data from data sources such as email servers, CRM systems, and customer support recording databases. The server retrieves this data through an API interface or by using a data collection agent and stores it in data storage. Users use their devices to upload data such as customer feedback and internal reports and send it to the server. The input of this step is raw data such as emails, voice recordings, and images, and the output is the collected, unprocessed data.

[1225] Step 2: Data Preprocessing

[1226] The server preprocesses the collected raw data. For audio data, it converts it into text using speech recognition software (e.g., a speech recognition API). For image data, it uses image processing software (e.g., OpenCV) to standardize the resolution and perform noise reduction if necessary. Furthermore, it normalizes the text data using a natural language processing toolkit (e.g., NLTK) to remove unnecessary characters and spaces. The input of this step is the raw data, and the output is the preprocessed data.

[1227] Step 3: Data analysis

[1228] The server analyzes the preprocessed data. Text data is semantically analyzed using natural language processing tools (e.g., spaCy) to perform sentiment analysis and build topic models. Image data and video data are analyzed using computer vision techniques (e.g., TensorFlow) to extract the necessary information. The analysis results include the data's topics, sentiment scores, and the detection of specific events and objects. The input to this step is the preprocessed data, and the output is the analysis results.

[1229] Step 4: Optimize business processes

[1230] The server generates optimization proposals for business processes based on the results of data analysis. For example, if many complaints with low sentiment scores are concentrated around a specific product, the server identifies the cause and proposes countermeasures (e.g., FAQ updates, product modifications). The user can review these proposals on their device and approve or modify them as necessary. The input to this step is the analysis results, and the output is optimization proposals.

[1231] Step 5: Document Management

[1232] The server uses an automated document management system (e.g., a document management platform) to efficiently manage internal and external documents. It converts uploaded documents into text using OCR technology (e.g., optical character recognition tools), classifies them based on categories, and stores them in relevant folders. Users can access the documents they need using a search interface. The input of this step is document data, and the output is classified and stored documents.

[1233] Step 6: Automate customer interactions

[1234] The terminal receives customer inquiries in real time and immediately sends the content to the server. The server uses natural language understanding technology (e.g., a language understanding model) to analyze the inquiry and generate an appropriate automated response. For simple inquiries, an automated response is sent to the customer immediately, but for complex inquiries, the user can review and modify the response. The input to this step is the text of the customer inquiry, and the output is the generated response.

[1235] (Application example 1)

[1236] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1237] In manufacturing processes within factories, the collection and analysis of a wide variety of data (text, audio, images, video) is not being carried out efficiently, making it difficult to optimize business processes and not fully automating document management and customer support. This makes it difficult to improve business efficiency and customer satisfaction. To solve this problem, an automated system that uses integrated data management and advanced analysis technology is required.

[1238] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1239] In this invention, the server includes a means for collecting text, audio, image, and video data, a means for preprocessing the collected data and converting it into a standard format, and a means for analyzing the preprocessed data and proposing business process optimization. This enables data analysis and optimization proposals to be made using a clustering algorithm in a manufacturing environment. The server also includes a means for converting audio data into text data and analyzing it, and a means for collecting and preprocessing data from various sensors, cameras, and audio recorders. Furthermore, by incorporating a means for analyzing the sentiment of audio data using natural language processing technology and improving the efficiency of customer service, it is possible to improve not only business efficiency but also customer satisfaction.

[1240] "Text data" refers to character string information written in a natural language, and includes formats such as documents, emails, and reports.

[1241] "Voice data" refers to digitally recorded human speech or acoustic signals, and is used for speech recognition and emotion analysis.

[1242] "Image data" is visual information stored in digital form, including photographs, graphics, and diagrams.

[1243] "Moving image data" refers to data that digitally records visual information that changes over time, including video clips and video recordings.

[1244] "Preprocessing" refers to a series of operations to convert collected data into a format suitable for analysis, including format standardization and noise removal.

[1245] A "clustering algorithm" is a machine learning technique for classifying data into multiple clusters, based on patterns and similarities in the data.

[1246] "Natural language processing" is a technology that enables computers to understand and analyze human language, and is used for semantic and sentiment analysis of text.

[1247] "Sentiment analysis" is a technology that detects and classifies emotions from text and voice data, and is useful for automating customer responses.

[1248] A "sensor" is a device that detects the physical state of an environment or object and is used to collect data.

[1249] A "camera" is a device that captures visual information and stores it as digital data, and is used to collect image data and video data.

[1250] "Audio recording device" means a device for recording audio in digital form and is used to collect speech and acoustic data.

[1251] This invention is an AI-driven automation system that aims to improve the efficiency of work processes in a factory. In an embodiment of the invention, the system is composed of three entities: a server, a terminal, and a user.

[1252] System Configuration

[1253] The server plays the primary role of collecting data, preprocessing, analyzing, optimizing, managing, and responding to customers. The terminal is a device that inputs and outputs data and provides a user interface, and is also used for robots installed on production lines in factories. The user operates the system, uploads data, and checks and approves proposed results.

[1254] Program processing overview

[1255] Data collection

[1256] The server periodically collects text data, audio data, image data, and video data from various sensors, cameras, and audio recorders in the factory. For example, the server acquires this data from the production line's recording devices and quality control systems. Users use their terminals to upload data such as production daily reports and error logs.

[1257] Data Preprocessing

[1258] The server preprocesses the collected data. For example, it converts audio data to text data and standardizes the resolution of image data. Text data is normalized and unnecessary characters and spaces are removed. This ensures that the data is in a consistent format and suitable for analysis.

[1259] Data analysis

[1260] The preprocessed data is then analyzed by the server. Natural language processing is used to perform semantic and sentiment analysis of the text data and build a topic model. A clustering algorithm is also used to classify the manufacturing data into clusters. Image and video data is analyzed using computer vision technology to extract the necessary information. This allows issues in the manufacturing process within the factory to be identified.

[1261] Business process optimization

[1262] Based on the analysis results, the server generates business process optimization proposals. For example, if multiple manufacturing errors occur for the same reason, the server will identify the cause and propose improvement measures. Users can check the proposals on their devices and approve or modify them as necessary.

[1263] Document Management

[1264] The server uses an automatic document management system to efficiently manage documents inside and outside the factory. Uploaded documents are automatically categorized and placed in relevant folders. Users can easily access the documents they need using tag or keyword searches.

[1265] Automating customer interactions

[1266] The terminal receives inquiries from customers and immediately sends them to the server, which uses natural language understanding technology to analyze the inquiries and generate appropriate responses. Responses are automatically sent, and users can review and modify them as needed.

[1267] Specific examples

[1268] As a concrete example, when quality control is performed on a manufacturing line in a factory, the system operates as follows.

[1269] 1. The server periodically collects new data from the production line's sensor data and quality control records.

[1270] 2. The server preprocesses these data and converts them into a standard format.

[1271] 3. The server analyzes the data using natural language processing and clustering algorithms to identify key manufacturing challenges and frequent problems.

[1272] 4. Based on the analysis results, the server identifies where problems are concentrated in specific manufacturing processes and generates improvement proposals.

[1273] 5. The user checks the proposed improvements on the device and approves them if necessary.

[1274] 6. Manufacturing data is automatically categorized and saved in relevant folders for future reference.

[1275] Prompt Sentence Examples

[1276] "Collect new complaint emails from the support mailbox, analyze the content of the complaint using natural language processing, and generate a report of the results."

[1277] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1278] Step 1:

[1279] Data collection

[1280] The server periodically collects text data, voice data, image data, and video data from various sensors, cameras, and audio recorders in the factory. For example, the server receives inputs from the production line monitoring system and quality control system to acquire these data, and outputs the collected data to be stored in the centralized management system.

[1281] Step 2:

[1282] Data Preprocessing

[1283] The server preprocesses the collected data. For example, it converts audio data into text data and standardizes the resolution of image data. Specifically, it uses voice recognition software to convert audio data into text and image processing algorithms to standardize the resolution of images. This preprocessing ensures that the data is in a consistent format, providing appropriate input data for analysis. The output is standardized text, audio, image, and video data.

[1284] Step 3:

[1285] Data analysis

[1286] The preprocessed data is then analyzed by the server, which uses natural language processing to perform semantic and sentiment analysis of the text data. For example, a generative AI model is used to detect sentiment from the text data and build a topic model. A clustering algorithm is also used to classify the manufacturing data into clusters. Image and video data is analyzed using computer vision techniques to extract information for anomaly detection and quality assessment on the production line. The output of this step is a detailed report based on the analysis results.

[1287] Step 4:

[1288] Business process optimization

[1289] Based on the analysis results, the server generates optimization proposals for business processes. The server identifies patterns of manufacturing errors and frequent problems and proposes specific improvement measures. It also uses a customized generative AI model to generate optimization proposals. These optimization proposals include specific instructions and action plans for improving the manufacturing process, thereby enabling efficient business operations. The output is a proposal for improving the business process.

[1290] Step 5:

[1291] Document Management

[1292] The server uses an automatic document management system to efficiently manage documents inside and outside the factory. Uploaded documents are automatically categorized and placed in relevant folders. They are also designed to be easily accessible using tag and keyword searches, allowing users to quickly access the documents they need. This improves the efficiency of document management. The output is categorized and organized document data.

[1293] Step 6:

[1294] Automating customer interactions

[1295] The terminal receives an inquiry from a customer and immediately sends the content to the server. The server uses natural language understanding technology to analyze the content of the inquiry and generate an appropriate response. The generated response is sent automatically, but the user can confirm and modify it if necessary. A prompt sentence (e.g., "Please collect new complaint emails from the support mailbox, analyze the content of the complaint using natural language processing, and generate a report of the results") may also be used to generate an initial response. The output of this step is a prompt and appropriate response to improve customer satisfaction.

[1296] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1297] This invention is an AI-driven automation system aimed at improving the efficiency of corporate operations, and in particular, by combining it with an emotion engine that recognizes user emotions, it achieves more advanced optimization of business processes and automation of customer support. In the embodiments of the present invention, a method is described for integrating and analyzing text, audio, image, and video data to automate optimization of business processes, document management, and customer support that take user emotions into consideration.

[1298] System Configuration

[1299] The system mainly consists of three components: a server, a terminal, and a user. The server is primarily responsible for data collection, preprocessing, analysis, optimization, management, customer service, and emotion recognition. The terminal is a device that inputs and outputs data and provides a user interface. The user operates the system, uploads data, and reviews and approves proposal results.

[1300] Program processing overview

[1301] Data collection

[1302] The server collects data from various sources, including email servers, CRM systems, and customer support recording databases, periodically retrieving text, audio, image, and video data. Users use their devices to upload data, such as customer feedback and internal reports, to the system.

[1303] Data Preprocessing

[1304] The server preprocesses the collected data and converts it into a standard format. For example, audio data is transcribed, image data is standardized, and text data is normalized to remove unnecessary characters and whitespace. This ensures the data is in a consistent format and suitable for analysis.

[1305] Data Analysis and Emotion Recognition

[1306] The preprocessed data is then analyzed by the server. Natural language processing is used to perform semantic analysis of the text data, and sentiment analysis is also performed simultaneously. For example, positive, negative, and neutral emotions are extracted from the text data. Voice data is also analyzed using a sentiment extraction engine to recognize the customer's emotions. Image and video data are also analyzed using computer vision technology to identify emotions from the user's facial expressions.

[1307] Business process optimization

[1308] Based on the analysis results and the recognized emotions, the server generates business process optimization proposals. For example, if many customers have negative emotions about a particular product, the server will identify the cause and propose improvements. The user can review the proposals on their device and approve or modify them as necessary.

[1309] Document Management

[1310] The server uses an automatic document management system to efficiently manage internal and external documents. Uploaded documents are automatically categorized and placed in relevant folders. Users can easily access the documents they need using tag or keyword searches.

[1311] Automating customer interactions

[1312] The terminal receives inquiries from customers and immediately sends them to the server. The server uses natural language understanding technology to analyze the inquiries and generate appropriate responses. Responses can also be tailored based on the user's sentiment. Responses are automatically sent, and users can review and modify them as needed.

[1313] Specific examples

[1314] For example, if a company receives similar complaints from multiple customers at its support desk, the system works as follows:

[1315] 1. The server periodically collects new complaint emails from the support mailbox.

[1316] 2. The server preprocesses the email data and normalizes the text data.

[1317] 3. The server uses natural language processing to analyze the content of the complaint and extract key topics and frequently occurring keywords.

[1318] 4. At the same time, the server uses an emotion engine to recognize customer emotions from the text data. For example, if many emails express negative emotions, this is detected.

[1319] 5. Based on the analysis results and sentiment data, the server identifies problems concentrated in specific products or services and generates improvement proposals. For example, if there is a lot of negative feedback stating that "there are problems with the quality of a specific product," it will propose improvement measures for that product.

[1320] 6. The user checks the proposed improvements on the terminal and approves them if necessary.

[1321] 7. Complaint data will be automatically categorized and saved in relevant folders for future reference.

[1322] 8. When a new inquiry comes in from a customer, the server generates an automatic response and quickly replies to the customer with a message that reflects the user's feelings.

[1323] By implementing this invention, enterprises can significantly improve their business efficiency and provide high-quality customer service that takes into consideration users' emotions.

[1324] The processing flow will be explained below.

[1325] Step 1:

[1326] The server collects data from various sources: email data from the mail server, customer interaction data from the CRM system, and voice data from the recording database. The server periodically checks these data sources and collects any new data.

[1327] Step 2:

[1328] Users upload data to the system using their devices. Specifically, they drag and drop customer feedback audio files and image data into a dedicated upload interface and send them to the server. Users do this as part of their daily work.

[1329] Step 3:

[1330] The server preprocesses the collected data. For example, it transcribes audio data and standardizes the resolution of image data. It normalizes text data and removes unnecessary characters and whitespace. This preprocessing ensures that the data is in a consistent format and suitable for analysis.

[1331] Step 4:

[1332] The server analyzes the preprocessed data. Specifically, it uses natural language processing (NLP) to perform semantic analysis of the text data. It uses Python NLP libraries (e.g., spaCy, NLTK) to analyze the structure of the text and build sentiment analysis and topic models. It also uses speech recognition and emotion recognition engines to extract emotions from audio data.

[1333] Step 5:

[1334] The server analyzes image and video data. It uses computer vision technology (e.g., OpenCV, TensorFlow) to analyze the image data and identify emotions from the user's facial expressions. Similarly, it extracts emotions from video data based on specific facial expressions and movements.

[1335] Step 6:

[1336] The server uses an emotion recognition engine to extract the user's emotion from the analyzed data. For text data, it calculates an emotion score such as positive, negative, or neutral, and generates similar emotion scores for audio and image data.

[1337] Step 7:

[1338] The server generates business process optimization proposals based on the analysis results and sentiment data. For example, if many customers have negative feelings about a particular product, the server identifies the cause and proposes improvement measures. The proposals are generated in the form of specific action plans and changes.

[1339] Step 8:

[1340] The user checks the business process optimization proposals from the server through their terminal. For example, they review the proposals displayed on the terminal and approve or adjust them as necessary. If the user is satisfied with the proposals, they click the approval button to put them into action.

[1341] Step 9:

[1342] The server automatically classifies and manages documents. For example, it automatically analyzes uploaded documents such as contracts and customer feedback documents, attaches relevant tags, and moves them to the appropriate folders, improving document search efficiency.

[1343] Step 10:

[1344] Users search for and access the documents they need. Specifically, they use the device's search function to enter keywords or tags to quickly identify the desired document. They then select the relevant document from the search results and access it to obtain the information they need.

[1345] Step 11:

[1346] The terminal receives inquiries from customers, analyzes the content using natural language understanding (NLU) technology, and sends the inquiry to a server. For example, this is done by chatbots and email reception interfaces.

[1347] Step 12:

[1348] The server generates an appropriate response based on the inquiry. Specifically, it searches for similar cases using past inquiry data and a knowledge base, and generates an automatic response. The generated response is quickly sent to the customer.

[1349] Step 13:

[1350] The user checks the automated response generated by the server, makes any necessary corrections, and then sends the final reply. For example, by checking the response text on the terminal interface, correcting it, and then clicking the send button, the user can provide accurate information to the customer.

[1351] Example 2

[1352] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1353] Modern business activities require the efficient management and analysis of massive amounts of data, as well as the optimization of business processes and the automation of customer responses. In particular, an approach that handles text, voice, image, and video data in an integrated manner and takes user emotions into account is essential, but there are currently no systems that can effectively achieve this. Furthermore, managing and analyzing data manually is time-consuming, costly, and prone to human error. Therefore, there is a need for a comprehensive system that automates everything from data collection and analysis to optimization proposals, document management, and customer responses.

[1354] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1355] In this invention, the server includes means for collecting text, audio, image, and video data from various data sources, means for preprocessing the collected data and converting it into a standard format, and means for analyzing the preprocessed data using a natural language processing and emotion recognition engine. This enables companies to efficiently manage and analyze massive amounts of data, optimize business processes, and automate customer responses.

[1356] "Various data sources" refers to multiple information sources, including email servers, customer relationship management systems, customer support recording databases, etc.

[1357] "Text data" refers to data expressed in text format, such as emails, reports, and chat history.

[1358] "Audio data" refers to data expressed in the form of voice, such as audio files and recorded data.

[1359] "Image data" refers to data expressed in a visual format, such as photographs, screenshots, and illustrations.

[1360] "Video data" refers to data that is presented in a moving visual format, such as a video clip or recorded data.

[1361] "Collect" refers to the act of obtaining data and putting it into a system.

[1362] "Preprocessing" refers to the initial processing of collected data to convert it into a form that is easier to analyze. Examples include transcribing audio data and normalizing text data.

[1363] "Standard format" refers to a standard for standardizing data into a consistent format.

[1364] "Natural language processing" refers to the technology that enables computers to understand and analyze natural language written and spoken by humans.

[1365] An "emotion recognition engine" refers to software or algorithms that identify and analyze emotions contained within data.

[1366] "Analyzing" refers to the act of analyzing data to understand its content and patterns and extract information.

[1367] "Business process optimization" refers to the act of planning and implementing proposals and measures to improve the efficiency of corporate activities.

[1368] "Automated document management" refers to the automatic performance of management tasks such as document classification, storage, and retrieval by a computer system.

[1369] "Customer response automation" refers to a system automatically generating and sending appropriate responses to customer inquiries.

[1370] This invention is an AI-driven automation system aimed at improving business efficiency, and in particular, by combining an emotion engine that recognizes user emotions, it optimizes business processes and automates customer support. This system is mainly composed of three components: a server, a terminal, and a user. Specific embodiments of this system are described below.

[1371] System Configuration

[1372] The system consists of three components: a server, a terminal, and a user. The server is primarily responsible for data collection, preprocessing, analysis, optimization, management, customer support, and emotion recognition. The terminal is a device that inputs and outputs data and provides a user interface. The user operates the system, uploads data, and confirms and approves proposal results. Specific implementation examples of each component are shown below.

[1373] server

[1374] The server periodically collects text, audio, image, and video data from various data sources (e.g., email servers, CRM systems, customer support recording databases, etc.). The collected data is ingested through an ETL (Extract, Transform, Load) process. The server then preprocesses the collected data. Audio data is transcribed using a speech-to-text engine (e.g., Google Cloud Speech-to-Text), image data is standardized in resolution, and video data is broken down into frames. Text data is normalized, and unnecessary characters and whitespace are removed. Regular expressions and natural language processing tools (e.g., NLTK) are used in this process.

[1375] The preprocessed data is then analyzed using a natural language processing (NLP) engine (e.g., BERT or GPT model) to perform semantic analysis of the text data. A sentiment analysis engine (e.g., Sentiment140) is used to extract positive, negative, or neutral sentiment. Audio data is also analyzed using a sentiment extraction engine. Finally, image and video data are analyzed using computer vision techniques (e.g., OpenCV or deep learning models) to identify emotions from the user's facial expressions.

[1376] Based on the analysis results, the server generates business process optimization proposals. For example, if many customers have negative feelings about a particular product, it will suggest improvements for that product. It also uses an automated document management system to efficiently manage uploaded documents. Documents are categorized into appropriate folders, and necessary documents can be quickly accessed using tag or keyword searches.

[1377] Terminal

[1378] The terminal is a device that allows users to input and output data and provides a user interface. Users upload data through the terminal and review and approve proposals and analysis results. When users upload data, the terminal checks the data format and displays a warning if the format is inappropriate. The terminal also receives inquiries from customers and immediately sends the details to the server.

[1379] User

[1380] Users operate the system to upload data and review and approve proposal results. Specifically, they use terminals to upload data such as customer feedback and internal reports to the system. They also review the proposal content and approve or modify it if necessary. When a customer inquiry is received, they can also review the generated response and make modifications as necessary.

[1381] Specific examples

[1382] For example, if a company receives similar complaints from multiple customers at its support desk, the system works as follows:

[1383] 1. The server periodically collects new complaint emails from the support mailbox.

[1384] 2. The server preprocesses the email data and normalizes the text data.

[1385] 3. The server uses natural language processing to analyze the content of the complaint and extract key topics and frequently occurring keywords.

[1386] 4. At the same time, the server uses an emotion engine to recognize customer emotions from the text data. For example, if many emails express negative emotions, this is detected.

[1387] 5. Based on the analysis results and sentiment data, the server identifies problems concentrated in specific products or services and generates improvement proposals. For example, if there is a lot of negative feedback stating that "there are problems with the quality of a specific product," it will propose improvement measures for that product.

[1388] 6. The user checks the proposed improvements on the terminal and approves them if necessary.

[1389] 7. Complaint data will be automatically categorized and saved in relevant folders for future reference.

[1390] 8. When a new inquiry comes in from a customer, the server generates an automatic response and quickly replies to the customer with a message that reflects the user's feelings.

[1391] Prompt Sentence Examples

[1392] Below are some examples of text data to analyze and specific prompts to extract sentiment:

[1393] Prompt: "Conduct a customer feedback analysis and generate a report that extracts key topics and their respective sentiments. Analyze the following feedback: 'Product X is difficult to use. Support is slow to respond.'"

[1394] By feeding this prompt into a generative AI model, the system can analyze the main topics of the feedback (e.g., usability, support response) and sentiment (negative) and generate a corresponding report.

[1395] In this way, the present invention significantly improves the business efficiency of companies, and in particular enables high-quality responses that take into account the user's emotions.

[1396] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1397] Step 1:

[1398] The server collects text, audio, image, and video data from various data sources (e.g., email servers, CRM systems, customer support recording databases, etc.) and periodically ingests new data by running an ETL (Extract, Transform, Load) process. This step uses data from various data sources as input and produces the collected data as output.

[1399] Step 2:

[1400] The server preprocesses the collected data. For example, audio data is transcribed using a speech-to-text engine, image data is standardized in resolution, and video data is broken down into frames. Text data is normalized using regular expressions and natural language processing tools. This step uses the collected raw data as input and produces preprocessed data as output.

[1401] Step 3:

[1402] The server analyzes the preprocessed data. It uses a natural language processing engine (e.g., a GPT model) to perform semantic analysis of the text data and a sentiment analysis engine to extract positive, negative, or neutral sentiment. Audio data is also analyzed using a sentiment extraction engine, and image and video data are analyzed using computer vision techniques. This step uses the preprocessed data as input and generates analysis results as output.

[1403] Step 4:

[1404] The server generates business process optimization proposals based on the analysis results. For example, if a particular product has received a lot of negative feedback, it generates improvement proposals for that product. It also includes automated processes for customer support. This step uses the analysis results as input and generates optimization proposals and automated responses as output.

[1405] Step 5:

[1406] The user uses a terminal to review the improvement proposals and the automated response. The terminal displays the proposals and provides an interface for the user to approve or modify them as needed. If the user makes modifications, the modified proposals are fed back into the system. This step uses the optimization proposals and automated response as inputs and generates the proposals that have been reviewed and modified by the user as output.

[1407] Step 6:

[1408] The server uses an automated document management system to classify and store the uploaded documents. The documents are categorized into appropriate folders and can be efficiently accessed using tag and keyword searches. This step uses the uploaded documents as input and produces classified and stored documents as output.

[1409] Step 7:

[1410] When the terminal receives a customer inquiry, it immediately sends the content to the server, which analyzes the inquiry and generates an appropriate response using natural language understanding technology. The response can also be tailored based on the user's sentiment. This step uses the customer inquiry information as input and provides the generated response as output.

[1411] (Application example 2)

[1412] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1413] In conventional corporate business processes, it was difficult to analyze user emotions in real time and provide optimal responses. Furthermore, when it came to displaying advertisements, static ads were often displayed without considering user emotions, making it impossible to deliver advertisements effectively. This resulted in problems such as lower customer satisfaction and reduced business efficiency.

[1414] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1415] In this invention, the server includes means for collecting text, audio, image, and video data, means for preprocessing the collected data and converting it into a standard format, means for analyzing the preprocessed data and proposing business process optimization, means for automated document management, means for automating customer support, means for tracking user gaze and analyzing emotions in real time, and means for displaying optimal advertisements based on the analyzed emotions. This allows business processes and advertisement display to be optimized based on user emotions, enabling highly efficient and high-quality customer support.

[1416] "Text data" refers to data that contains sentences or character information, and includes emails, chat messages, documents, and the like.

[1417] "Voice data" means data obtained by recording a human voice or other sounds, including voice messages, telephone recordings, and customer support audio recordings.

[1418] "Image data" refers to visual information recorded as still images, including photographs, screenshots, illustrations, etc.

[1419] "Movie data" refers to dynamic visual information consisting of a series of image frames, including video clips and video recordings.

[1420] "Collection methods" are the technical means used to obtain text, audio, image, and video data from a variety of sources, including database access and use of APIs.

[1421] "Preprocessing means" refers to technology that converts collected data into a format that is easier to analyze, such as transcribing audio data or standardizing image resolution.

[1422] A "standard format" is a format in which data of different formats is converted into a consistent format that can be analyzed, making data analysis easier.

[1423] "Means of analysis" refers to technologies that use preprocessed data to extract information and optimize or propose business processes, including machine learning and natural language processing.

[1424] "Means for proposing optimization" refers to techniques that propose methods and means for improving the efficiency of business processes based on the analysis results, and includes formulating plans and presenting improvement measures.

[1425] "Means for document management" refers to technology for efficiently managing documents, including automatic classification and tagging of documents, and search functions.

[1426] "Means for automating customer support" refers to technology that automatically responds to inquiries and requests from customers, and includes chatbots and automatic reply systems.

[1427] "Gaze tracking means" refers to technology that detects the direction of a user's gaze and monitors it in real time, and includes gaze tracking cameras and sensors.

[1428] "Means for analyzing emotions in real time" refers to technology that detects the user's emotions at that time from their facial expressions, voice, and text, and includes facial expression recognition engines and emotion analysis algorithms.

[1429] The "means for displaying advertisements" refers to technology that displays appropriate advertisements to users based on the analyzed emotions, and includes displays and marketing engines.

[1430] This invention is an AI-driven automation system aimed at improving business efficiency and optimizing customer service. The system includes means for collecting text, voice, image, and video data, means for preprocessing the collected data and converting it into a standard format, means for analyzing the preprocessed data and proposing business process optimization, means for automated document management, and means for automating customer service. It also includes means for tracking a user's gaze and analyzing their emotions in real time, and means for displaying optimal advertisements based on the analyzed emotions.

[1431] System Configuration

[1432] This system is mainly composed of three components: a server, a terminal, and a user. The server is primarily responsible for data collection, preprocessing, analysis, optimization, emotion recognition, and display of advertisements. The terminal is a device that inputs and outputs data and provides a user interface. The user operates the system, uploads data, and reviews and approves the proposed results.

[1433] Data collection

[1434] The server collects text, audio, image, and video data from various data sources, such as email servers, CRM systems, and customer support recording databases, and users use their devices to upload data to the system, including customer feedback and internal reports.

[1435] Data Preprocessing

[1436] The server preprocesses the collected data and converts it into a standard format. For example, audio data is transcribed, image data is standardized, and text data is normalized to remove unnecessary characters and whitespace. This ensures the data is in a consistent format and suitable for analysis.

[1437] Data Analysis and Emotion Recognition

[1438] The preprocessed data is then analyzed by the server. Natural language processing is used to perform semantic analysis of the text data, and sentiment analysis is also performed simultaneously. For example, positive, negative, and neutral emotions are extracted from the text data. Voice data is also analyzed using a sentiment extraction engine to recognize customer emotions. Image and video data are also analyzed using computer vision technology to identify emotions from the user's facial expressions.

[1439] Business process optimization

[1440] Based on the analysis results and the recognized emotions, the server generates business process optimization proposals. For example, if many customers have negative emotions about a particular product, the server will identify the cause and propose improvements. The user can review the proposals on their device and approve or modify them as necessary.

[1441] Document Management

[1442] The server efficiently manages internal and external documents using an automatic document management system. Uploaded documents are automatically categorized and placed in relevant folders. Users can easily access the documents they need using tag or keyword searches.

[1443] Automating customer interactions

[1444] The terminal receives inquiries from customers and immediately sends them to the server. The server uses natural language understanding technology to analyze the inquiries and generate appropriate responses. It can also adjust the responses based on the user's sentiment. Responses are automatically sent, and users can review and modify them as needed.

[1445] Eye Tracking and Emotion Analysis

[1446] When a user wears smart glasses, the eye tracking camera detects the direction of their gaze and analyzes their facial expressions in real time. The server collects this data and performs emotion analysis. Based on the analyzed emotion, the server selects the most appropriate advertisement and displays it in the user's field of view.

[1447] Specific examples

[1448] When a user wears smart glasses and browses web content, Serve analyzes their gaze and facial expressions in real time. For example, if a user shows interest in a particular product page, Serve can detect positive emotions and display relevant ads based on those emotions. This maximizes the effectiveness of advertising and improves the user experience.

[1449] An example of a prompt is, "Design an application that recognizes emotions in real time from the gaze and facial expressions captured by the camera when the user is wearing smart glasses, and displays the most appropriate advertisement based on that emotion. Specifically, implement a function that uses emotion recognition technology to detect the main emotion and selects and displays an appropriate advertisement image. In addition, the advertisement image will be selected from a list that has been pre-classified into several emotional categories."

[1450] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1451] Step 1: Data collection

[1452] The server periodically retrieves text, audio, image, and video data from mail servers, CRM systems, and customer support recording databases. Users use their devices to upload data, such as customer feedback and internal reports, to the system. The input is various data from external data sources, and the output is the collected raw data. Specifically, the server retrieves data using API calls and database queries.

[1453] Step 2: Preprocessing the data

[1454] The server preprocesses the collected data and converts it into a standard format. Specifically, it transcribes audio data, standardizes the resolution of image data, and normalizes text data to remove unnecessary characters and spaces. The input is the raw data collected, and the output is the preprocessed data. The server performs these conversions using a speech recognition engine and image processing algorithms.

[1455] Step 3: Data analysis and emotion recognition

[1456] The server analyzes the preprocessed data and uses natural language processing to perform semantic analysis of the text data, while simultaneously performing sentiment analysis. For example, positive, negative, and neutral emotions are extracted from the text data. Voice data is analyzed using an emotion extraction engine to recognize customer emotions. Image data and video data are analyzed using computer vision technology. The input is the preprocessed data, and the output is the analysis results and emotional data. Specifically, the server performs these analyses using a natural language processing library and a facial recognition engine.

[1457] Step 4: Optimize business processes

[1458] The server generates optimization proposals for business processes based on the analysis results and the recognized emotions. For example, if many customers have negative emotions about a particular product, the cause can be identified and improvement measures proposed. The input is the analysis results and emotion data, and the output is optimization proposals for business processes. Specifically, the server generates optimization proposals using machine learning algorithms.

[1459] Step 5: Document Management

[1460] The server uses an automated document management system to efficiently manage internal and external documents. Uploaded documents are automatically categorized and placed in relevant folders. Users can access the documents they need using tag or keyword searches. The input is the uploaded document data, and the output is organized and categorized documents. Specifically, the server uses document management software.

[1461] Step 6: Automate customer interactions

[1462] The terminal receives an inquiry from a customer and immediately sends the content to the server. The server uses natural language understanding technology to analyze the inquiry and generate an appropriate response. The input is the customer's inquiry, and the output is the generated response message. Specifically, the server uses a natural language understanding engine.

[1463] Step 7: Gaze Tracking and Emotion Analysis

[1464] When a user wears smart glasses, an eye-tracking camera is used to detect the direction of gaze and analyze facial expressions in real time. The server collects this data and performs emotion analysis. The input is the user's gaze and facial expression data, and the output is analyzed emotion data. Specifically, the server uses an eye-tracking camera and facial expression recognition software.

[1465] Step 8: Optimizing Ad Display

[1466] The server selects the most suitable advertisement based on the analyzed emotion and displays it in the user's field of view. The input is the analyzed emotion data, and the output is the displayed advertisement image. Specifically, the server uses an advertisement selection algorithm and a display device.

[1467] An example of a prompt is, "Design an application that recognizes emotions in real time from the gaze and facial expressions captured by the camera when the user is wearing smart glasses, and displays the most appropriate advertisement based on that emotion. Specifically, implement a function that uses emotion recognition technology to detect the main emotion and selects and displays an appropriate advertisement image. In addition, the advertisement image will be selected from a list that has been pre-classified into several emotional categories."

[1468] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1469] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1470] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1471] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1472] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper...

Claims

1. means for collecting text, audio, image, and video data; a means of preprocessing the collected data and converting it into a standard format; A means of analyzing pre-processed data and proposing business process optimization; a means for performing automated document management; A means of automating customer responses; A system including:

2. 10. The system of claim 1, further comprising means for analyzing text, audio, image, and video data using multimodal AI.

3. 10. The system of claim 1, further comprising means for performing semantic analysis of the text data using natural language processing.

Images