System

Abstract

A system is provided.SOLUTION: A system comprising: means for identifying faces of meeting participants; means for generating a list of the identified meeting participants; means for converting questions asked by the meeting participants into text in real time using speech recognition technology; means for retrieving and generating materials and information related to the questions; and means for categorizing issues discussed during the meeting and presenting analysis results and solutions.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditionally, large-scale meetings have often been moderated by a human, which requires time for preparation and makes it difficult to provide appropriate answers and materials on the spot. Furthermore, manual recording and organization of issues often results in inefficient follow-up after the meeting. The present invention aims to solve these problems and improve the efficiency and quality of meetings. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for identifying the faces of meeting participants, a means for generating a list of identified meeting participants, a means for converting the participants' questions into text in real time using voice recognition technology, a means for searching for and generating materials and information related to the questions, and a means for categorizing issues discussed during the meeting and presenting analysis results and solutions. The system also includes a means for analyzing the content discussed after the meeting and generating prediction results, allowing for efficient post-meeting follow-up. By automatically identifying meeting participants using a face recognition algorithm and utilizing voice recognition technology and a database, it is possible to provide appropriate materials and information on the spot.

[0006] "Conference Participant" means an individual present at or participating remotely in a Conference.

[0007] A "facial identification device" is a combination of hardware and software used to capture images and recognize and match specific faces.

[0008] The "means for generating a list" is a system for collecting information on recognized conference participants and creating a list in a list format.

[0009] "Speech recognition technology" refers to technology for converting voice data into text, and includes, for example, a voice recognition engine.

[0010] "Means for converting to text" refers to the process of converting voice data into text information using voice recognition technology.

[0011] "Means for retrieving and generating materials and information" refers to the process for retrieving the necessary materials and information from databases and other sources and presenting them to conference participants.

[0012] A "categorization tool" is a system and algorithm for separating the issues discussed into specific categories.

[0013] The "Method of Presenting Analysis Results and Solutions" is the process for analyzing data on the issues discussed and providing solutions and their results.

[0014] A "means for generating predicted results" is a process for predicting and presenting future results based on past and current data.

[0015] A "face recognition algorithm" is an algorithm that recognizes faces using specific features, and includes, for example, Haar Cascade and MTCNN. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] This invention provides an automatic conference announcer system that recognizes the faces of conference participants and processes questions in real time. This system manages the conference from start to finish, instantly generating and providing necessary materials and information, and also analyzes issues and presents solutions after the conference ends.

[0038] Overall system configuration

[0039] The system comprises three main components: a server, a terminal, and a user.

[0040] 1. Server

[0041] The server is the heart of the system and has the following main functions:

[0042] Facial recognition: The faces of meeting participants are identified from camera footage and compared with a pre-registered database.

[0043] Question content analysis: The user's question is converted into text using voice recognition technology, and the content is analyzed.

[0044] Material search: Search and generate the necessary materials and information from the database.

[0045] Issue analysis: Analyze the data accumulated during the meeting and classify and analyze the issues discussed by category.

[0046] Solution Proposal: Based on the analysis results, we provide future solutions and predicted results.

[0047] 2. Terminal

[0048] The terminal is an input device for in-room and remote participants and has the following main functions:

[0049] Camera and microphone: Captures video and audio of meeting participants and sends them to the server.

[0050] Display device: Displays materials and information sent from the server on the conference screen.

[0051] 3. Users

[0052] A user is an individual who participates in a conference and performs the following actions:

[0053] Voice questions: Ask voice questions during the meeting.

[0054] Viewing materials: View materials and information displayed on the device.

[0055] Program processing explanation

[0056] The processing of the programs in this system is mainly carried out in cooperation between the server, the terminal, and the user.

[0057] Facial recognition and participant list creation

[0058] 1. The device captures video of the conference room with its camera and sends it to the server.

[0059] 2. The server uses a facial recognition algorithm to identify attendees' faces and compare them with a registered database. Based on the results of the comparison, it generates an attendee list, which is sent to the device and displayed on the screen.

[0060] Real-time question processing and answer generation

[0061] 1. The user speaks a question, and the device captures the audio and sends it to the server.

[0062] 2. The server uses voice recognition technology to convert the question into text and analyzes the content.

[0063] 3. The server searches the database for materials and information related to the question, generates the necessary materials, and sends them to the terminal to display on the screen.

[0064] Post-conference analysis and solution presentation

[0065] 1. The server analyzes the data (audio, text, materials) accumulated during the meeting and extracts the content discussed.

[0066] 2. The server classifies the extracted issues into categories and performs detailed analysis.

[0067] 3. The server generates solutions and prediction results for each problem, compiles them into a report, and sends it to the terminal to share with the meeting participants.

[0068] Specific examples

[0069] For example, consider a case where a company holds a meeting to discuss marketing strategies for a new product.

[0070] Step 1: When the meeting starts, the device captures the video of the meeting room, and the server identifies all attendees and displays a list of participants.

[0071] Step 2: When a user asks, "What is the target market for the new product?" the device captures the voice, the server converts it into text, and generates and displays relevant marketing materials.

[0072] Step 3: After the meeting, the server analyzes the content discussed and generates a report presenting issues and solutions, such as "reviewing marketing strategies" and "optimizing advertising budgets," which is shared with meeting participants via their devices.

[0073] In this way, the present invention allows for the provision of appropriate information in real time and efficient meeting follow-up.

[0074] The processing flow will be explained below.

[0075] Step 1:

[0076] The device captures video data from within the conference room using a camera and transmits it to the server in real time.

[0077] Step 2:

[0078] The server receives the captured video data and applies a facial recognition algorithm to identify the faces of the meeting participants. Specifically, it uses a face detection model (e.g., Haar Cascade or MTCNN) to identify facial features, then extracts and matches them with face data in a database.

[0079] Step 3:

[0080] The server generates a list of identified conference participants based on the matching result and transmits the generated list to the terminal.

[0081] Step 4:

[0082] The device displays the list of conference participants received on the screen. For example, "Yamada Taro - Project Manager" or "Sato Hanako - Engineer."

[0083] Step 5:

[0084] The user speaks a question, for example, "What is the target market for the new product?"

[0085] Step 6:

[0086] The terminal captures the voice data of the question with a microphone device and transmits the captured voice data to the server.

[0087] Step 7:

[0088] The server converts the voice data into text using a speech recognition engine (e.g., Google® Cloud Speech-to-Text).

[0089] Step 8:

[0090] The server analyzes the textual question using natural language processing (NLP) technology, for example, to extract keywords such as "target market" from the question.

[0091] Step 9:

[0092] The server searches the database and generates materials and information related to the query (for example, presentation slides on marketing strategies).

[0093] Step 10:

[0094] The server transmits the generated materials to the terminal, which then displays them on the conference screen.

[0095] Step 11:

[0096] After the meeting ends, the server analyzes the data (audio, text, and materials) accumulated during the meeting.

[0097] Step 12:

[0098] Based on the analyzed data, the server categorizes the issues discussed, such as "reviewing marketing strategies" or "optimizing advertising budgets."

[0099] Step 13:

[0100] The server performs a detailed analysis of each category and considers solutions based on relevant data.

[0101] Step 14:

[0102] The server compiles solutions and prediction results for each problem into a report.

[0103] Step 15:

[0104] The server sends the generated report to the terminal and shares it with the conference participants through the terminal.

[0105] This series of steps enables efficient management of the meeting progress, provision of appropriate information in real time, and post-meeting analysis of issues and presentation of solutions.

[0106] Example 1

[0107] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0108] During a meeting, it is necessary to identify participants' faces, process questions in real time, search for and generate relevant materials, analyze the issues discussed, and present solutions. However, it is difficult to perform these tasks efficiently and automatically. It is also equally difficult to quickly analyze the content discussed after the meeting and present solutions afterward. Conventional systems only automate part of these processes, resulting in a decrease in overall efficiency.

[0109] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0110] In this invention, the server includes means for identifying the faces of conference participants, means for generating a list of identified conference participants, means for converting the contents of questions asked by the conference participants into text in real time using voice recognition technology, means for searching for and generating materials and information related to the contents of the questions, means for categorizing issues discussed during the conference and presenting analysis results and solutions, means for transmitting the generated materials and information to terminals and displaying them on the conference screen, and means for analyzing accumulated data after the conference ends, extracting and categorizing the contents discussed, and generating solutions. This makes it possible to automate and efficiently manage the overall process of conference progress and follow-up.

[0111] A "conference participant" is an individual participating in a conference.

[0112] A "means for identifying faces" is a technology or device for analyzing camera footage and recognizing the faces of individual conference participants.

[0113] The "means for generating a list of conference participants" is a technology or device that creates a list of people participating in a conference based on the results of face identification.

[0114] "Speech recognition technology" is a technology that converts voice data into text data in real time.

[0115] "Means for converting questions into text in real time using voice recognition technology" refers to a technology or device that captures questions uttered by conference participants as voice data and instantly converts it into text.

[0116] "Means for searching and generating materials and information" refers to technology or equipment that searches for necessary materials and information from a database and generates them on request.

[0117] A "means for categorizing issues" is a technique or device that separates the content discussed during a meeting into multiple categories.

[0118] "Means for presenting analysis results and solutions" refers to a technique or device that analyzes the content of the discussion and derives and presents solutions based on the results.

[0119] The "means for transmitting generated materials and information to the terminal" refers to a technology or device for transmitting various materials and information generated by the server to the terminal in the conference room.

[0120] The "means for displaying on the conference screen" refers to a technique or device for displaying materials and information sent to the terminal on the conference screen.

[0121] The "means for analyzing data accumulated after the end of a meeting" refers to a technology or device for analyzing data such as audio, text, and materials collected after the end of a meeting.

[0122] "Means for extracting and classifying the content discussed and generating solutions" refers to a technology or device that analyzes the content discussed during a meeting, extracts important elements, classifies them by category, and generates solutions.

[0123] This invention provides an automatic conference announcer system that recognizes the faces of conference participants and processes questions in real time. This system manages the conference from start to finish, instantly generating and providing necessary materials and information, and also analyzes issues and presents solutions after the conference ends.

[0124] Overall system configuration

[0125] The system comprises three main components: a server, a terminal, and a user.

[0126] 1. Server

[0127] The server is the heart of the system and has the following main functions:

[0128] Facial recognition: The server receives the video footage from the conference room camera sent from the device, identifies attendees using a facial recognition algorithm, and compares them with a registered database. Technologies used include OpenCV and Dlib.

[0129] Question analysis: The server receives the data captured by the device and sends the voice question from the user, and converts the question into text using speech recognition technology. This can be done using the Google Speech-to-Text API. The text question is then analyzed using natural language processing technology (e.g., SpaCy).

[0130] Material search: The server uses a search engine such as ElasticSearch (registered trademark) to search and generate the necessary materials and information from the database.

[0131] Issue analysis: The server analyzes the data accumulated during the meeting and categorizes and analyzes the issues discussed using natural language processing toolkits (e.g., NLTK, BERT).

[0132] Solution Proposal: The server generates future solutions and predictions based on the analysis results. In this process, it uses a generative AI model (e.g., GPT-4 (registered trademark)) to provide the optimal solution.

[0133] 2. Terminal

[0134] The terminal is an input device for in-room and remote participants and has the following main functions:

[0135] Camera and microphone: The device captures the video and audio of meeting participants in real time and sends them to the server.

[0136] Display device: The terminal receives materials and information sent from the server and displays them on the conference screen. This can be an electronic whiteboard or video conferencing software (e.g., Zoom, MICROSOFT® TEAMS®).

[0137] 3. Users

[0138] A user is an individual who participates in a conference and performs the following actions:

[0139] Voice questions: Users ask questions by voice during the conference, which are captured by the terminal and sent to the server.

[0140] Viewing materials: Users view materials and information displayed on their terminals and obtain appropriate information as the meeting progresses.

[0141] Program processing explanation

[0142] Facial recognition and participant list creation

[0143] The device captures video of the conference room with its camera and sends it to the server, where the server uses a facial recognition algorithm to identify attendees' faces and compare them with a registered database. Based on the results of the comparison, a list of attendees is generated and sent to the device, where it is displayed on the screen.

[0144] Real-time question processing and answer generation

[0145] The user speaks a question, and the device captures the voice and sends it to the server. The server uses voice recognition technology to convert the question into text and analyzes it. The server then searches a database for materials and information related to the question and generates the necessary materials. These are then sent to the device and displayed on the screen.

[0146] Post-conference analysis and solution presentation

[0147] The server analyzes the data (audio, text, and materials) accumulated during the meeting and extracts the content of the discussion. The server then categorizes the extracted issues and performs a detailed analysis. The server then generates solutions and predictions for each issue, compiles them into a report, and sends it to the device to share with the meeting participants.

[0148] Specific examples

[0149] For example, consider a case where a company holds a meeting to discuss marketing strategies for a new product.

[0150] Step 1: When the meeting starts, the device captures the video of the meeting room, and the server identifies all attendees and displays a list of participants.

[0151] Step 2: When a user asks, "What is the target market for the new product?" the device captures the voice, the server converts it into text, and generates and displays relevant marketing materials.

[0152] Step 3: After the meeting, the server analyzes the content discussed and generates a report presenting issues and solutions, such as "reviewing marketing strategies" and "optimizing advertising budgets," which is shared with meeting participants via their devices.

[0153] Examples of prompt statements

[0154] "Please display information about the target market for any new products discussed during the meeting. Also provide examples of past successful marketing strategies."

[0155] As described above, the present invention realizes appropriate information provision in real time and efficient conference follow-up.

[0156] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0157] Step 1:

[0158] The device captures video of the conference room with a camera and sends the data to the server in real time. This input data is a video stream, and each frame contains the faces of the conference participants. The server receives this data and performs preprocessing for face identification. Specifically, it analyzes the video data and detects and extracts faces from each frame.

[0159] Step 2:

[0160] The server applies a facial recognition algorithm (e.g., OpenCV or Dlib) to the preprocessed image data, extracts the features of the recognized face, and compares them with a pre-registered database. The matching results are output as the ID and name of the identified conference participant.

[0161] Step 3:

[0162] The server generates an attendee list based on the matching results. This list includes the name and related information of each conference participant. The generated attendee list is sent from the server to the terminal and displayed on the terminal's conference screen. This data output is displayed on the conference screen as text information.

[0163] Step 4:

[0164] The user speaks a question, and the device captures and sends the audio to the server. The input data is an audio file or a real-time audio stream. The server receives the data and performs preprocessing for speech recognition, such as noise reduction and speech normalization.

[0165] Step 5:

[0166] The server uses speech recognition technology (e.g., Google Speech-to-Text API) to convert the preprocessed voice data into text, and the output is the question in text format.

[0167] Step 6:

[0168] The server analyzes the text of the question using natural language processing (NLP) technology (e.g., SpaCy). As a result of this analysis, the intent of the question and related keywords are extracted.

[0169] Step 7:

[0170] The server searches the database for relevant materials and information based on the analysis results. This process uses a search engine such as Elasticsearch. The search results include links to related materials and summaries.

[0171] Step 8:

[0172] The server generates the necessary materials based on the search results and sends them to the terminal. The generated materials are displayed on the terminal's conference screen. This output data includes PDFs and presentation slides.

[0173] Step 9:

[0174] After the meeting ends, the server analyzes the accumulated meeting data (audio, text, and materials). It uses a natural language processing toolkit (e.g., NLTK, BERT) to extract and classify the issues discussed. The output is a list of the extracted issues and their categorization.

[0175] Step 10:

[0176] The server generates solutions to each problem based on the analysis results. Using a generative AI model (e.g., GPT-4), it outputs optimal solutions and predictions. These are sent to the terminal in the form of a report and shared with the meeting participants.

[0177] Step 11:

[0178] The device displays the received report to the meeting participants, allowing them to view the shared information, which allows for efficient follow-up after the meeting has ended.

[0179] In this way, the overall process of meeting progress and follow-up is detailed for each processing step.

[0180] (Application example 1)

[0181] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0182] Conventional conference systems have difficulty recognizing the faces of conference participants, processing questions in real time, generating materials instantly, and analyzing issues and presenting solutions after the conference has ended. Furthermore, in customer service at brick-and-mortar stores, there are challenges in effectively identifying customers' faces and providing personalized service, making it difficult to improve customer satisfaction. Furthermore, there is a need to analyze customer voices and instantly present in-store product information and inventory status. To solve these problems, a system capable of more efficient, real-time information processing and personalized response is needed.

[0183] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0184] In this invention, the server includes means for identifying the faces of conference participants, means for generating a list of identified conference participants, means for converting the content of questions from conference participants into text in real time using voice recognition technology, means for searching for and generating materials and information related to the content of the questions, means for categorizing issues discussed during the conference and presenting analysis results and solutions, means installed in smart glasses for identifying customers through entity recognition and providing personalized service, and means for analyzing the customer's voice and presenting product information and inventory status in the store. This enables appropriate information provision and efficient follow-up of issues in conferences, and enables real-time, personalized support to improve customer satisfaction in physical stores.

[0185] The "means for identifying the faces of conference participants" refers to a method of capturing the faces of conference participants using a camera and identifying individual participants using a facial recognition algorithm.

[0186] The "means for generating a list of identified conference participants" is a method for organizing and displaying information about conference participants identified by facial recognition in list form.

[0187] "Means for converting questions from conference participants into text in real time using speech recognition technology" refers to a method for converting speech spoken by conference participants into text data in real time using speech recognition technology.

[0188] "Means for searching and generating materials and information related to the question content" refers to a method for searching a database for related materials and information based on the analyzed question content, and generating and displaying them.

[0189] "Method of categorizing the issues discussed during the meeting and presenting the analysis results and solutions" is a method of organizing and analyzing the content discussed during the meeting by category, and presenting the analysis results and solutions based on that.

[0190] "Means installed on smart glasses for identifying customers through entity recognition and providing personalized service" refers to a method for identifying customers visiting a store using a camera and a recognition system installed in smart glasses and providing personalized service to the customer.

[0191] "Means for analyzing customer voice and presenting in-store product information and stock status" refers to a method for analyzing customer voice in real time and displaying and providing in-store product information and stock status based on the analysis results.

[0192] The present invention is a system including two main application examples: a conference system and a brick-and-mortar store system. This system is configured with a server, a terminal, and a user as its main components.

[0193] Conference System

[0194] server

[0195] The server has the following main functions:

[0196] 1. Facial recognition: The faces of meeting participants are identified from camera footage and compared with a pre-registered database. OpenCV and TENSORFLOW (registered trademark) are used for facial recognition.

[0197] 2. Speech recognition: The user's question is converted into text using PyAudio and SpeechRecognition technology, and the content is analyzed.

[0198] 3. Document search: Search and generate documents and information related to the question from the database. This is done using SQLAlchemy.

[0199] 4. Issue analysis: Analyze the data accumulated during the meeting and classify and analyze the issues discussed by category.

[0200] 5. Solution Proposal: Based on the analysis results, provide future solutions and predicted results.

[0201] Terminal

[0202] The terminal has the following main features:

[0203] 1. Camera and microphone: Captures video and audio of meeting participants and sends them to the server.

[0204] 2. Display device: Displays materials and information sent from the server on the conference screen.

[0205] User

[0206] A user is an individual who participates in a conference and performs the following actions:

[0207] 1. Audio Questions: Ask questions by voice during the meeting.

[0208] 2. Viewing materials: View materials and information displayed on the terminal.

[0209] Physical store compatible system

[0210] Smart Glasses

[0211] The smart glasses have the following main features:

[0212] 1. Facial Recognition: Recognize the faces of customers who visit your physical store and identify repeat and new customers. OpenCV and TensorFlow are used for facial recognition.

[0213] 2. Speech Recognition: Analyze customer questions in real time and provide appropriate information, using PyAudio and SpeechRecognition technology.

[0214] 3. Product information display: Display product information and stock status based on customer questions. Use SQLAlchemy for database management.

[0215] server

[0216] The server has the following main functions:

[0217] 1. Facial recognition: Identify customers using video transmitted from smart glasses.

[0218] 2. Speech recognition: Converts the customer's voice into text and analyzes its content.

[0219] 3. Information search: Search and generate in-store product information and inventory status based on customer questions.

[0220] 4. History management: Maintain customer visit and purchase history to provide personalized service.

[0221] Specific examples

[0222] For example, if a repeat customer visits a store, the camera in the smart glasses will recognize the customer and respond with a voice message saying, "Welcome. Thank you for visiting again." If the customer asks, "Is this item in stock?", the voice recognition system will respond, "The item in question is currently in stock."

[0223] Also, if a participant asks during a meeting, "Is there a recent report on this issue?", the server will convert the speech into text, search for relevant materials, and display them.

[0224] Prompt Sentence Examples

[0225] "Generate program code for a smart glasses assistant that recognizes customer faces, greets returning and new customers appropriately, and provides inventory information and product recommendations based on customer questions. Use OpenCV and TensorFlow for face recognition, PyAudio and SpeechRecognition for voice recognition, and SQLAlchemy for database management."

[0226] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0227] Step 1:

[0228] Facial Recognition Processing

[0229] Server: Receives video footage of the conference room or store from the device and extracts facial features using OpenCV. Next, it uses TensorFlow's facial recognition algorithm to calculate facial features and compare them with facial data in the database.

[0230] Input: Video data from the device.

[0231] Output: A list of recognized meeting participants or customers.

[0232] Specific operation: The camera detects faces in each captured video frame, calculates features, and matches them with face information in a matching database.

[0233] Step 2:

[0234] Generate and display a list of conference participants

[0235] Server: Generates a list of identified conference participants based on facial recognition and sends the list to the terminal.

[0236] Input: Face identification results.

[0237] Output: A list of conference participants.

[0238] Specific operation: The recognized participant's name or ID is added to a list, sent to the device, and displayed.

[0239] Step 3:

[0240] Capture and transcribing voice questions

[0241] Terminal: The microphone captures the user's voice and sends it to the server.

[0242] Server: Converts received audio data into text using PyAudio and SpeechRecognition technology.

[0243] Input: Audio data from the user.

[0244] Output: Textualized question.

[0245] Specific operation: The user's voice is detected by the microphone, sent to the server in real time, and converted into text using voice recognition technology.

[0246] Step 4:

[0247] Searching for and generating materials and information

[0248] Server: Analyzes the textual content of the question, searches for and generates relevant materials and information from the database using SQLAlchemy, and sends the materials to the terminal.

[0249] Input: The text of the question.

[0250] Output: Related materials and information.

[0251] Specific operation: The database is searched using the question as a keyword, and relevant documents and information are extracted and sent to the terminal.

[0252] Step 5:

[0253] Categorizing problems and presenting solutions

[0254] Server: Analyzes questions and discussions accumulated during the meeting and categorizes them. Generates solutions for each category and sends detailed analysis results to the device.

[0255] Input: Questions and discussions during the meeting.

[0256] Output: Categorised challenges and their solutions.

[0257] Specific operation: The accumulated data is analyzed using natural language processing technology and classified into categories, and then a solution is generated using an AI model and sent to the terminal in report format.

[0258] Step 6:

[0259] Identifying and responding to customers in-store

[0260] Smart glasses: Capture the face of the customer and send it to the server. Based on the results of the server's matching, the glasses present the customer with appropriate greetings and information.

[0261] Input: Customer video data.

[0262] Output: Customer identification and response message.

[0263] How it works: The smart glasses' camera captures the customer's face and sends it to the server. When the recognition result is returned, a voice or visual response is given based on that data.

[0264] Step 7:

[0265] Analysis of customer voice questions and information presentation

[0266] Smart glasses: Capture customer voice questions with a microphone and send them to a server.

[0267] Server: Analyzes the voice data, searches for and generates product information and stock status based on the question, and sends that information to the smart glasses.

[0268] Input: Voice data from the customer.

[0269] Output: Product information and stock status.

[0270] Specific operation: The customer's voice is captured by a microphone and sent to a server, where it is analyzed using voice recognition technology, and the appropriate information is searched for in a database and sent to and displayed on the smart glasses.

[0271] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0272] This system combines an automatic announcer that recognizes the faces of conference participants and processes questions in real time with an emotion engine that recognizes the user's emotions. This system supports the smooth progress of conferences, provides information according to the participants' emotions, and efficiently follows up after the conference.

[0273] Overall system configuration

[0274] The system mainly includes three main components and one additional component: a server, a terminal, a user, and an emotion engine.

[0275] 1. Server

[0276] The server is the center of the system and has the following main functions:

[0277] Facial recognition: The faces of meeting participants are identified from camera footage and compared with a pre-registered database.

[0278] Question content analysis: The user's question is converted into text using voice recognition technology, and the content is analyzed.

[0279] Material search: Search and generate the necessary materials and information from the database.

[0280] Issue analysis: Analyze the data accumulated during the meeting and classify and analyze the issues discussed by category.

[0281] Solution Proposal: Based on the analysis results, we provide future solutions and predicted results.

[0282] Sentiment data analysis: Receive data from the sentiment engine and use it for follow-up during and after the meeting.

[0283] 2. Terminal

[0284] The terminal is an input device for in-room and remote participants and has the following main functions:

[0285] Camera and microphone: Captures video and audio of meeting participants and sends them to the server.

[0286] Display device: Displays materials and information sent from the server on the conference screen.

[0287] Emotion recognition: Analyze participants' facial expressions and tone of voice to obtain emotional data.

[0288] 3. Users

[0289] A user is an individual who participates in a conference and performs the following actions:

[0290] Voice questions: Ask voice questions during the meeting.

[0291] Viewing materials: View materials and information displayed on the device.

[0292] 4. Emotion Engine

[0293] The emotion engine is a component that recognizes and analyzes the user's emotions.

[0294] Emotion analysis: Analyze user emotions in real time through facial expression recognition and voice analysis.

[0295] Data provision: Analysis results are sent to the server and used to dynamically change materials and information.

[0296] Program processing explanation

[0297] The system's programs are processed in cooperation with the server, terminal, user, and emotion engine.

[0298] Facial recognition and participant list creation

[0299] 1. The device captures video from within the conference room using a camera and transmits it to the server in real time.

[0300] 2. The server applies a facial recognition algorithm to identify the attendee's face and match it with a database.

[0301] 3. The server generates an attendee list based on the matching results and sends it to the terminal.

[0302] 4. The terminal displays the list of conference participants on the screen.

[0303] Real-time question processing and answer generation

[0304] 1. The user speaks a question, and the device captures the audio and sends it to the server.

[0305] 2. The server uses a speech recognition engine to convert the voice data into text and analyze the content.

[0306] 3. The server searches and generates relevant materials and information from the database.

[0307] 4. The server sends the generated materials to the terminal, which displays them on the conference screen.

[0308] Sentiment analysis and information provision

[0309] 1. The device captures the user's facial expressions and tone of voice using an emotion engine and analyzes them in real time.

[0310] 2. The emotion engine sends the acquired emotion data to the server.

[0311] 3. The server dynamically changes the information provided based on the emotion data and updates materials and information as necessary.

[0312] Post-conference analysis and solution presentation

[0313] 1. The server analyzes the data (voice, text, materials, emotional data) accumulated during the meeting.

[0314] 2. The server uses this data to categorize the issues discussed and conduct a detailed analysis.

[0315] 3. The server compiles solutions and prediction results for each problem into a report, sends it to the terminal, and shares it with the meeting participants.

[0316] Specific examples

[0317] For example, consider a case where a company holds a meeting to discuss marketing strategies for a new product.

[0318] Step 1: When the meeting starts, the device captures the video of the meeting room, and the server identifies all attendees and displays the participant list.

[0319] Step 2: When a user asks, "What is the target market for the new product?" the device captures the voice, the server converts it into text, and generates and displays relevant marketing materials.

[0320] Step 3: During the meeting, if the emotion engine senses the user's doubts or concerns, the server updates the materials and provides additional information.

[0321] Step 4: After the meeting ends, the server analyzes the discussion content and emotional data, generates a report presenting issues and solutions such as "reviewing marketing strategies" and "optimizing advertising budgets," and shares this report with meeting participants via their devices.

[0322] In this way, the present invention makes it possible to provide appropriate information in real time, dynamically change materials according to the emotions of participants, and also realizes the analysis of problems and the presentation of solutions after the meeting has ended.

[0323] The processing flow will be explained below.

[0324] Step 1:

[0325] The device captures video data from within the conference room using a camera and transmits it to the server in real time.

[0326] Step 2:

[0327] The server receives the captured video data and applies a facial recognition algorithm to identify the faces of the meeting participants. Specifically, it uses a face detection model (e.g., Haar Cascade or MTCNN) to identify facial features, then extracts and matches them with face data in a database.

[0328] Step 3:

[0329] The server generates a list of identified conference participants based on the matching result and transmits the generated list to the terminal.

[0330] Step 4:

[0331] The device displays the list of conference participants received on the screen. For example, "Yamada Taro - Project Manager" or "Sato Hanako - Engineer."

[0332] Step 5:

[0333] The user speaks a question, for example, "What is the target market for the new product?"

[0334] Step 6:

[0335] The terminal captures the voice data of the question with a microphone device and transmits the captured voice data to the server.

[0336] Step 7:

[0337] The server converts the voice data into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[0338] Step 8:

[0339] The server analyzes the textual question using natural language processing (NLP) technology, for example, to extract keywords such as "target market" from the question.

[0340] Step 9:

[0341] The server searches the database and generates materials and information related to the query (for example, presentation slides on marketing strategies).

[0342] Step 10:

[0343] The server transmits the generated materials to the terminal, which then displays them on the conference screen.

[0344] Step 11:

[0345] The device captures the user's facial expressions and tone of voice using an emotion engine and analyzes them in real time.

[0346] Step 12:

[0347] The emotion engine sends the acquired emotion data to the server. For example, if the user feels anxious or suspicious about a question, that emotion is sent to the server.

[0348] Step 13:

[0349] The server dynamically changes the information provided based on the emotion data, and searches and generates additional materials and information from the database as needed.

[0350] Step 14:

[0351] The server generates additional materials and sends them to the terminal, which then displays them on the conference screen. For example, if the user is feeling anxious, detailed statistical data or additional explanatory materials can be displayed.

[0352] Step 15:

[0353] After the meeting ends, the server analyzes the data accumulated during the meeting (voice, text, materials, and emotional data).

[0354] Step 16:

[0355] Based on the analyzed data, the server categorizes the issues discussed, such as "reviewing marketing strategies" or "optimizing advertising budgets."

[0356] Step 17:

[0357] The server performs a detailed analysis of each category and considers solutions based on relevant data.

[0358] Step 18:

[0359] The server compiles solutions and prediction results for each problem into a report.

[0360] Step 19:

[0361] The server sends the generated report to the terminal and shares it with the conference participants through the terminal. For example, after the conference, a report on the proposal for "optimizing the advertising budget" and its feasibility is shared.

[0362] This series of steps allows for efficient management of the progress of meetings, provision of appropriate information in real time, and post-meeting analysis of issues and presentation of solutions.It also enables dynamic changes to materials based on the emotions of participants.

[0363] Example 2

[0364] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0365] Conventional conference systems were able to convert participants' questions into text using facial recognition and voice recognition, and to search for and generate materials. However, they lacked the means to analyze participants' emotions in real time and dynamically update materials and information based on that. Another issue was the inability to provide predictive results through data analysis after the meeting. This could result in meetings not progressing smoothly and participants' satisfaction declining.

[0366] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for identifying the faces of the conference participants, means for generating a list of the identified conference participants, means for converting the content of the questions of the conference participants into text in real time using voice recognition technology, means for searching for and generating materials and information related to the content of the questions, means for capturing and analyzing emotional data of the conference participants, means for dynamically changing the information provided based on the emotional data and updating the materials and information as necessary, and means for categorizing issues discussed during the conference and presenting analysis results and solutions. This makes it possible to identify the faces and content of the questions of the conference participants as well as dynamically provide information according to the emotions of the participants, thereby realizing smooth progress of the conference and follow-up.

[0367] "Conference participant" refers to an individual who attends a conference and is a user of the system.

[0368] "Means for identifying faces" refers to a device or process that uses a facial recognition algorithm to identify conference participants based on camera footage.

[0369] A "facial recognition algorithm" is a set of mathematical formulas and programs that use image processing technology to recognize and identify individual faces.

[0370] "Means for generating a list of conference participants" refers to a device or process that organizes information about identified conference participants and displays or records it in list form.

[0371] "Voice recognition technology" is a technology that analyzes human voice and converts the content into text data.

[0372] "Means for converting questions into text" refers to a device or process that uses speech recognition technology to convert voice questions from conference participants into text data in real time.

[0373] "Means for retrieving and generating materials and information" refers to a device or process that retrieves the required materials and information from a database and generates them in an appropriate format.

[0374] "Means for capturing and analyzing emotional data" refers to a device or process that uses a camera or microphone to capture the facial expressions and tone of voice of meeting participants and analyzes them using an emotion analysis algorithm.

[0375] "Means for dynamically changing information provision based on emotional data" refers to a device or process that updates or changes the materials or information provided in real time based on analyzed emotional data.

[0376] "Means for categorizing issues and presenting analysis results and solutions" refers to a device or process that analyzes data from the content discussed during a meeting, categorizes it, and then proposes the analysis results and solutions based on them.

[0377] "Means for generating prediction results" refers to a device or process that analyzes the data accumulated after the meeting and generates results to predict future developments.

[0378] MODE FOR CARRYING OUT THE INVENTION

[0379] This system combines an automatic announcer that recognizes the faces of conference participants and processes their questions in real time with an emotion engine that recognizes and analyzes user emotions. This system supports the smooth progress of conferences, provides information according to the participants' emotions, and efficiently follows up after the conference.

[0380] Overall system configuration

[0381] The system mainly includes three main components and one additional component: a server, a terminal, a user, and an emotion engine.

[0382] 1. Server

[0383] The server is the center of the system and has the following main functions:

[0384] Facial Identification: Uses OpenCV to identify the faces of meeting participants from camera footage and match them with a pre-registered database.

[0385] Question content analysis: The user's question is converted into text using voice recognition technology (Google Cloud Speech-to-Text), and the content is analyzed.

[0386] Material search: Search and generate the necessary materials and information from the database.

[0387] Issue analysis: Analyze the data accumulated during the meeting and classify and analyze the issues discussed by category.

[0388] Solution Proposal: Based on the analysis results, we provide future solutions and predicted results.

[0389] Sentiment data analysis: Receive data from the sentiment engine and use it for follow-up during and after the meeting.

[0390] 2. Terminal

[0391] The terminal is an input device for in-room and remote participants and has the following main functions:

[0392] Camera and microphone: Captures video and audio of meeting participants and sends them to the server.

[0393] Display device: Displays materials and information sent from the server on the conference screen.

[0394] Emotion recognition: Analyze participants' facial expressions and tone of voice to obtain emotional data.

[0395] 3. Users

[0396] A user is an individual who participates in a conference and performs the following actions:

[0397] Voice questions: Ask voice questions during the meeting.

[0398] Viewing materials: View materials and information displayed on the device.

[0399] 4. Emotion Engine

[0400] The emotion engine is a component that recognizes and analyzes the user's emotions.

[0401] Emotion Analysis: Uses the Microsoft Azure® Emotion API to analyze user emotions in real time through facial expression recognition and voice analysis.

[0402] Data provision: Analysis results are sent to the server and used to dynamically change materials and information.

[0403] Specific examples

[0404] For example, consider a case where a company holds a meeting to discuss marketing strategies for a new product.

[0405] Step 1: At the start of the meeting, the device captures the video of the meeting room, and the server identifies all attendees. A participant list is created and displayed on the device.

[0406] Step 2: When a user asks, "What is the target market for the new product?", the device captures the voice, the server converts the voice into text, and generates and displays related materials.

[0407] Step 3: During the meeting, if the emotion engine senses the user's doubts or concerns, the server updates the materials and provides additional information.

[0408] Step 4: After the meeting ends, the server analyzes the discussion content and emotional data, generates a report presenting issues and solutions such as "reviewing marketing strategies" and "optimizing advertising budgets," and shares this report with meeting participants via their devices.

[0409] These functions are important for ensuring smooth progress in meetings and providing appropriate information according to the emotions of participants.

[0410] Prompt Sentence Examples

[0411] Describe how your system will respond if a user asks, "What is the target market for the new product?"

[0412] As described above, the present invention makes it possible to provide appropriate information in real time, dynamically change materials in response to the emotions of participants, and also to analyze issues and present solutions after the meeting has ended.

[0413] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0414] Step 1: Capture footage

[0415] Input: Camera video sent from the device.

[0416] Processing: The device captures video in real time through the camera in the conference room and sends the data to the server.

[0417] Output: Captured video data.

[0418] Step 2: Facial Recognition and Matching

[0419] Input: Captured video data.

[0420] Processing: The server runs a facial recognition algorithm using OpenCV to identify participants' faces from the video data and match them with pre-registered facial data in a database.

[0421] Output: A list of identified conference participants.

[0422] Step 3: Generate a participant list

[0423] Input: A list of identified conference participants.

[0424] Processing: The server generates this list and sends it to the terminal.

[0425] Output: Participant list sent to the device.

[0426] Step 4: View the Participant List

[0427] Input: The participant list sent to the device.

[0428] Processing: The terminal displays the received participant list on the conference screen. For example, names such as "Taro Tanaka" and "Hanako Sato" are displayed in the list.

[0429] Output: Participant list displayed on the conference screen.

[0430] Step 5: Capture audio

[0431] Input: User's spoken question.

[0432] Processing: During a meeting, a user asks a question by voice, such as "Where are the marketing materials?" The device captures the voice with its microphone and sends it to the server as audio data.

[0433] Output: The audio data sent to the server.

[0434] Step 6: Speech to Text

[0435] Input: The audio data sent to the server.

[0436] Processing: The server uses Google Cloud Speech-to-Text to convert the audio data into text. For example, the audio is converted into text, such as "Where are the marketing materials?"

[0437] Output: The question converted to text.

[0438] Step 7: Parsing the Question

[0439] Input: The question converted to text.

[0440] Processing: The server analyzes the textual question and searches the database for relevant information and materials.

[0441] Output: Search results for related materials.

[0442] Step 8: Find and generate relevant materials

[0443] Input: Search results for related materials.

[0444] Processing: The server generates the necessary materials based on the search results, for example preparing a PDF file of marketing materials.

[0445] Output: The generated material.

[0446] Step 9: Viewing the material

[0447] Input: Generated materials.

[0448] Processing: The server sends the generated materials to the terminal, which displays them on the conference screen. The user can view the displayed materials.

[0449] Output: Materials displayed on the meeting screen.

[0450] Step 10: Capturing Emotional Data

[0451] Input: Facial expressions and tone of voice of meeting participants.

[0452] Processing: The device uses a camera and microphone to capture emotional data such as the user's facial expressions and tone of voice in real time.

[0453] Output: Captured emotion data.

[0454] Step 11: Analyze the emotion data

[0455] Input: Captured emotion data.

[0456] Processing: An emotion engine (e.g., Microsoft Azure Emotion API) analyzes the captured data and determines the user's emotion. For example, it outputs an analysis result such as "The user looks anxious."

[0457] Output: Parsed emotion data.

[0458] Step 12: Dynamic information provision and documentation updates

[0459] Input: Parsed emotion data.

[0460] Processing: The emotion engine sends the analysis results to the server. The server then provides dynamic information based on the emotion data, adapting to the situation. For example, if the user appears anxious, the server provides additional materials with additional explanations. The device then displays the latest materials on the conference screen.

[0461] Output: Updated documentation and information.

[0462] Step 13: Data collection and analysis

[0463] Input: Audio, text, materials, emotional data, etc. accumulated during the meeting.

[0464] Processing: The server centralizes this data and performs detailed analysis.

[0465] Output: Analysis results.

[0466] Step 14: Categorize and analyze the issue

[0467] Input: Analysis results.

[0468] Processing: Based on the analysis results, the server categorizes the issues discussed and conducts a detailed analysis.

[0469] Output: A list of issues sorted by category.

[0470] Step 15: Generate and share your report

[0471] Input: A list of issues sorted by category.

[0472] Processing: The server generates a report that describes the solutions and prediction results for each problem. The report is sent to the terminal and can be viewed by the meeting participants.

[0473] Output: Report shared with meeting participants.

[0474] (Application example 2)

[0475] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0476] Modern factories are required to immediately solve problems faced by workers and improve productivity. However, conventional systems lack real-time problem-solving capabilities and support that adapts to the emotions of workers. As a result, work efficiency declines and work errors become more likely to occur. The present invention aims to provide a real-time problem-solving assistant system specialized for factory environments to solve these problems.

[0477] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for identifying the faces of factory workers, means for generating a list of identified factory workers, means for converting the contents of questions from factory workers into text in real time using voice recognition technology, means for searching and generating procedures and information related to the contents of the questions, and means for classifying problems that occur during factory work into categories and presenting analysis results and solutions. This makes it possible to quickly solve problems faced by factory workers and improve work efficiency.

[0478] "Conference participant" refers to any person participating in a conference.

[0479] "Means for identifying faces" refers to technology that uses a camera or facial recognition algorithm to recognize the face of a target person and identify that person.

[0480] The "means for generating a list of identified conference participants" is a technology for creating a list of participants based on information about conference participants obtained by face recognition.

[0481] "Speech recognition technology" is a technology that converts human speech into text data in real time.

[0482] "Means for searching and generating materials and information" refers to technology that searches for the necessary materials and information from related databases and information sources based on the content of a user's question and provides it.

[0483] "Means of categorizing issues and presenting analysis results and solutions" is a technique for organizing and classifying problems and issues discussed during meetings or work, and proposing solutions to them.

[0484] "Factory worker" refers to a worker who is in charge of work in a factory.

[0485] "Means for identifying the faces of factory workers" refers to technology that uses cameras and facial recognition algorithms to recognize the faces of target workers and identify them.

[0486] The "means for generating a list of identified factory workers" is a technology for creating a list of on-site workers based on information about factory workers obtained through facial recognition.

[0487] "Means for converting the content of questions posed by factory workers into text in real time using voice recognition technology" refers to technology for converting the voices spoken by factory workers into text in real time.

[0488] The "means for searching and generating procedures and information related to the content of a question" is a technology that searches a database for and provides relevant work procedures and information based on the content of a question posed by a factory worker.

[0489] "Means of categorizing problems that occur during factory work and presenting analysis results and solutions" is a technique for organizing and classifying troubles and problems that occur during work and proposing solutions to them.

[0490] This invention is a real-time problem-solving assistant system for improving work efficiency in a factory environment. This system mainly consists of a server, a terminal, a user, and an emotion engine.

[0491] Server Features

[0492] The server is the center of the system and has the following important functions:

[0493] Facial recognition: Identify the faces of factory workers from camera footage and compare them with a pre-registered database. Uses the dlib library and OpenCV as facial recognition algorithms.

[0494] Question content analysis: The questions from the workers are converted into text using speech recognition technology, and the content is analyzed. This is done using the speech_recognition library.

[0495] Document search: Search and generate the necessary procedures and information from a database that stores work procedures and equipment information within the factory.

[0496] Problem analysis: Analyze the data accumulated during the work and classify and analyze the problems that arise by category.

[0497] Solution Proposal: Based on the analysis results, we provide future solutions and predicted results.

[0498] Emotion data analysis: Receive data from the emotion engine and use it for follow-up during and after work.

[0499] Device Features

[0500] The terminal is an input device used by factory workers and has the following functions:

[0501] Camera and microphone: Captures video and audio of factory workers and sends them to a server.

[0502] Display device: The procedure and information sent from the server are displayed on the work screen. Display devices include smart glasses and smartphones.

[0503] Emotion Recognition: Analyzes the facial expressions and tone of voice of workers to obtain emotional data. Emotion recognition is performed using the Affectiva API.

[0504] User Actions

[0505] The user refers to a factory worker and performs the following actions:

[0506] Voice questions: Ask questions by voice as you work.

[0507] View Instructions: View instructions and information displayed on your device.

[0508] Emotion Engine Functions

[0509] The emotion engine is a component that recognizes and analyzes the emotions of workers.

[0510] Emotion analysis: Analyze workers' emotions in real time through facial expression recognition and voice analysis.

[0511] Data provision: The analysis results are sent to the server and used to dynamically change procedures and information.

[0512] Specific examples

[0513] For example, consider a situation where a problem occurs during the operation of a machine in a factory.

[0514] Step 1: When work begins, the device captures the camera image, and the server identifies the worker. A list of workers is displayed.

[0515] Step 2: When the worker asks, "Why is the machine stuck?" the device captures the voice, the server transcribes it, and generates the relevant repair instructions.

[0516] Step 3: If the emotion engine senses confusion or anxiety from the worker during the work, the server updates the displayed repair procedure with additional information to support the worker.

[0517] Step 4: After the work is completed, the server analyzes the accumulated data and generates a report that presents solutions such as "machine maintenance methods" and "future improvements," which is shared with the worker via their terminal.

[0518] Prompt Sentence Examples

[0519] "Devant a system that describes the situation when a maintenance issue occurs in the factory and generates real-time solutions to specific questions. Also include the ability to understand worker sentiment and provide follow-up accordingly."

[0520] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0521] Step 1:

[0522] Input: A factory worker activates smart glasses and camera footage is captured.

[0523] Processing (data processing and calculation): The device (smart glasses) captures the video of the factory worker and sends it to the server. The server uses OpenCV and dlib libraries to perform facial recognition and compares it with existing data in the database.

[0524] Output: A list of identified factory workers is generated and sent to the terminal.

[0525] How it works: The server applies a facial recognition algorithm to identify factory workers, creates a list, and displays it on the device.

[0526] Step 2:

[0527] Input: A factory worker asks a voiceover, "Why is the machine stuck?"

[0528] Processing (data processing and calculation): The device captures the voice and sends it to the server. The server converts the voice data into text using the speech_recognition library and analyzes the question.

[0529] Output: Repair procedures and information related to the query are searched for and generated from the database.

[0530] Specific operation: The server searches the database based on the text of the question, and sends relevant repair procedures and information to the terminal, where they are displayed.

[0531] Step 3:

[0532] Input: As the factory worker continues to follow the displayed steps, the emotion engine captures their facial expressions and tone of voice.

[0533] Processing (data processing and calculation): The device captures emotion data and sends it to the emotion engine in real time. The emotion engine analyzes emotions using the Affectiva API.

[0534] Output: The captured emotion data is sent to a server, which provides additional information or assistance if needed.

[0535] Specific operation: The server receives the sentiment analysis results and provides further support information if the worker is experiencing difficulties.

[0536] Step 4:

[0537] Input: The worker completes the repair procedure and the data on the work is sent from the terminal to the server.

[0538] Processing (data processing and calculation): The server analyzes the data (voice, text, emotion data) accumulated during the work and categorizes the problems that arise. Based on the analysis results, it generates solutions and predictions.

[0539] Output: A report containing the analysis results and solutions is generated and sent to the device.

[0540] Specific operation: The server generates a report and sends it to the terminal to be shared with workers and managers.

[0541] Step 5:

[0542] Input: Server-generated reports are reviewed by operators and administrators.

[0543] Processing (data processing and calculation): The report is displayed on the terminal, and suggestions for improvements and maintenance methods for the next work are made as needed.

[0544] Output: Next work plan and improvement suggestions are provided as reference for workers and managers.

[0545] Specific action: Workers and managers use the report to plan the next work.

[0546] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0547] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0548] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0549] [Second embodiment]

[0550] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0551] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0552] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0553] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0554] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0555] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0556] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0557] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0558] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0559] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0560] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0561] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0562] This invention provides an automatic conference announcer system that recognizes the faces of conference participants and processes questions in real time. This system manages the conference from start to finish, instantly generating and providing necessary materials and information, and also analyzes issues and presents solutions after the conference ends.

[0563] Overall system configuration

[0564] The system comprises three main components: a server, a terminal, and a user.

[0565] 1. Server

[0566] The server is the heart of the system and has the following main functions:

[0567] Facial recognition: The faces of meeting participants are identified from camera footage and compared with a pre-registered database.

[0568] Question content analysis: The user's question is converted into text using voice recognition technology, and the content is analyzed.

[0569] Material search: Search and generate the necessary materials and information from the database.

[0570] Issue analysis: Analyze the data accumulated during the meeting and classify and analyze the issues discussed by category.

[0571] Solution Proposal: Based on the analysis results, we provide future solutions and predicted results.

[0572] 2. Terminal

[0573] The terminal is an input device for in-room and remote participants and has the following main functions:

[0574] Camera and microphone: Captures video and audio of meeting participants and sends them to the server.

[0575] Display device: Displays materials and information sent from the server on the conference screen.

[0576] 3. Users

[0577] A user is an individual who participates in a conference and performs the following actions:

[0578] Voice questions: Ask voice questions during the meeting.

[0579] Viewing materials: View materials and information displayed on the device.

[0580] Program processing explanation

[0581] The processing of the programs in this system is mainly carried out in cooperation between the server, the terminal, and the user.

[0582] Facial recognition and participant list creation

[0583] 1. The device captures video of the conference room with its camera and sends it to the server.

[0584] 2. The server uses a facial recognition algorithm to identify attendees' faces and compare them with a registered database. Based on the results of the comparison, it generates an attendee list, which is sent to the device and displayed on the screen.

[0585] Real-time question processing and answer generation

[0586] 1. The user speaks a question, and the device captures the audio and sends it to the server.

[0587] 2. The server uses voice recognition technology to convert the question into text and analyzes the content.

[0588] 3. The server searches the database for materials and information related to the question, generates the necessary materials, and sends them to the terminal to display on the screen.

[0589] Post-conference analysis and solution presentation

[0590] 1. The server analyzes the data (audio, text, materials) accumulated during the meeting and extracts the content discussed.

[0591] 2. The server classifies the extracted issues into categories and performs detailed analysis.

[0592] 3. The server generates solutions and prediction results for each problem, compiles them into a report, and sends it to the terminal to share with the meeting participants.

[0593] Specific examples

[0594] For example, consider a case where a company holds a meeting to discuss marketing strategies for a new product.

[0595] Step 1: When the meeting starts, the device captures the video of the meeting room, and the server identifies all attendees and displays a list of participants.

[0596] Step 2: When a user asks, "What is the target market for the new product?" the device captures the voice, the server converts it into text, and generates and displays relevant marketing materials.

[0597] Step 3: After the meeting, the server analyzes the content discussed and generates a report presenting issues and solutions, such as "reviewing marketing strategies" and "optimizing advertising budgets," which is shared with meeting participants via their devices.

[0598] In this way, the present invention allows for the provision of appropriate information in real time and efficient meeting follow-up.

[0599] The processing flow will be explained below.

[0600] Step 1:

[0601] The device captures video data from within the conference room using a camera and transmits it to the server in real time.

[0602] Step 2:

[0603] The server receives the captured video data and applies a facial recognition algorithm to identify the faces of the meeting participants. Specifically, it uses a face detection model (e.g., Haar Cascade or MTCNN) to identify facial features, then extracts and matches them with face data in a database.

[0604] Step 3:

[0605] The server generates a list of identified conference participants based on the matching result and transmits the generated list to the terminal.

[0606] Step 4:

[0607] The device displays the list of conference participants received on the screen. For example, "Yamada Taro - Project Manager" or "Sato Hanako - Engineer."

[0608] Step 5:

[0609] The user speaks a question, for example, "What is the target market for the new product?"

[0610] Step 6:

[0611] The terminal captures the voice data of the question with a microphone device and transmits the captured voice data to the server.

[0612] Step 7:

[0613] The server converts the voice data into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[0614] Step 8:

[0615] The server analyzes the textual question using natural language processing (NLP) technology, for example, to extract keywords such as "target market" from the question.

[0616] Step 9:

[0617] The server searches the database and generates materials and information related to the query (for example, presentation slides on marketing strategies).

[0618] Step 10:

[0619] The server transmits the generated materials to the terminal, which then displays them on the conference screen.

[0620] Step 11:

[0621] After the meeting ends, the server analyzes the data (audio, text, and materials) accumulated during the meeting.

[0622] Step 12:

[0623] Based on the analyzed data, the server categorizes the issues discussed, such as "reviewing marketing strategies" or "optimizing advertising budgets."

[0624] Step 13:

[0625] The server performs a detailed analysis of each category and considers solutions based on relevant data.

[0626] Step 14:

[0627] The server compiles solutions and prediction results for each problem into a report.

[0628] Step 15:

[0629] The server sends the generated report to the terminal and shares it with the conference participants through the terminal.

[0630] This series of steps enables efficient management of the meeting progress, provision of appropriate information in real time, and post-meeting analysis of issues and presentation of solutions.

[0631] Example 1

[0632] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0633] During a meeting, it is necessary to identify participants' faces, process questions in real time, search for and generate relevant materials, analyze the issues discussed, and present solutions. However, it is difficult to perform these tasks efficiently and automatically. It is also equally difficult to quickly analyze the content discussed after the meeting and present solutions afterward. Conventional systems only automate part of these processes, resulting in a decrease in overall efficiency.

[0634] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0635] In this invention, the server includes means for identifying the faces of conference participants, means for generating a list of identified conference participants, means for converting the contents of questions asked by the conference participants into text in real time using voice recognition technology, means for searching for and generating materials and information related to the contents of the questions, means for categorizing issues discussed during the conference and presenting analysis results and solutions, means for transmitting the generated materials and information to terminals and displaying them on the conference screen, and means for analyzing accumulated data after the conference ends, extracting and categorizing the contents discussed, and generating solutions. This makes it possible to automate and efficiently manage the overall process of conference progress and follow-up.

[0636] A "conference participant" is an individual participating in a conference.

[0637] A "means for identifying faces" is a technology or device for analyzing camera footage and recognizing the faces of individual conference participants.

[0638] The "means for generating a list of conference participants" is a technology or device that creates a list of people participating in a conference based on the results of face identification.

[0639] "Speech recognition technology" is a technology that converts voice data into text data in real time.

[0640] "Means for converting questions into text in real time using voice recognition technology" refers to a technology or device that captures questions uttered by conference participants as voice data and instantly converts it into text.

[0641] "Means for searching and generating materials and information" refers to technology or equipment that searches for necessary materials and information from a database and generates them on request.

[0642] A "means for categorizing issues" is a technique or device that separates the content discussed during a meeting into multiple categories.

[0643] "Means for presenting analysis results and solutions" refers to a technique or device that analyzes the content of the discussion and derives and presents solutions based on the results.

[0644] The "means for transmitting generated materials and information to the terminal" refers to a technology or device for transmitting various materials and information generated by the server to the terminal in the conference room.

[0645] The "means for displaying on the conference screen" refers to a technique or device for displaying materials and information sent to the terminal on the conference screen.

[0646] The "means for analyzing data accumulated after the end of a meeting" refers to a technology or device for analyzing data such as audio, text, and materials collected after the end of a meeting.

[0647] "Means for extracting and classifying the content discussed and generating solutions" refers to a technology or device that analyzes the content discussed during a meeting, extracts important elements, classifies them by category, and generates solutions.

[0648] This invention provides an automatic conference announcer system that recognizes the faces of conference participants and processes questions in real time. This system manages the conference from start to finish, instantly generating and providing necessary materials and information, and also analyzes issues and presents solutions after the conference ends.

[0649] Overall system configuration

[0650] The system comprises three main components: a server, a terminal, and a user.

[0651] 1. Server

[0652] The server is the heart of the system and has the following main functions:

[0653] Facial recognition: The server receives the video footage from the conference room camera sent from the device, identifies attendees using a facial recognition algorithm, and compares them with a registered database. Technologies used include OpenCV and Dlib.

[0654] Question analysis: The server receives the data captured by the device and sends the voice question from the user, and converts the question into text using speech recognition technology. This can be done using the Google Speech-to-Text API. The text question is then analyzed using natural language processing technology (e.g., SpaCy).

[0655] Material search: The server uses a search engine such as Elasticsearch to search and generate the necessary materials and information from the database.

[0656] Issue analysis: The server analyzes the data accumulated during the meeting and categorizes and analyzes the issues discussed using natural language processing toolkits (e.g., NLTK, BERT).

[0657] Solution Proposal: The server generates future solutions and predictions based on the analysis results, using a generative AI model (e.g., GPT-4) to provide the optimal solution.

[0658] 2. Terminal

[0659] The terminal is an input device for in-room and remote participants and has the following main functions:

[0660] Camera and microphone: The device captures the video and audio of meeting participants in real time and sends them to the server.

[0661] Display device: The terminal receives materials and information sent from the server and displays them on the conference screen. This can be an electronic whiteboard or video conferencing software (e.g., Zoom, Microsoft Teams).

[0662] 3. Users

[0663] A user is an individual who participates in a conference and performs the following actions:

[0664] Voice questions: Users ask questions by voice during the conference, which are captured by the terminal and sent to the server.

[0665] Viewing materials: Users view materials and information displayed on their terminals and obtain appropriate information as the meeting progresses.

[0666] Program processing explanation

[0667] Facial recognition and participant list creation

[0668] The device captures video of the conference room with its camera and sends it to the server, where the server uses a facial recognition algorithm to identify attendees' faces and compare them with a registered database. Based on the results of the comparison, a list of attendees is generated and sent to the device, where it is displayed on the screen.

[0669] Real-time question processing and answer generation

[0670] The user speaks a question, and the device captures the voice and sends it to the server. The server uses voice recognition technology to convert the question into text and analyzes it. The server then searches a database for materials and information related to the question and generates the necessary materials. These are then sent to the device and displayed on the screen.

[0671] Post-conference analysis and solution presentation

[0672] The server analyzes the data (audio, text, and materials) accumulated during the meeting and extracts the content of the discussion. The server then categorizes the extracted issues and performs a detailed analysis. The server then generates solutions and predictions for each issue, compiles them into a report, and sends it to the device to share with the meeting participants.

[0673] Specific examples

[0674] For example, consider a case where a company holds a meeting to discuss marketing strategies for a new product.

[0675] Step 1: When the meeting starts, the device captures the video of the meeting room, and the server identifies all attendees and displays a list of participants.

[0676] Step 2: When a user asks, "What is the target market for the new product?" the device captures the voice, the server converts it into text, and generates and displays relevant marketing materials.

[0677] Step 3: After the meeting, the server analyzes the content discussed and generates a report presenting issues and solutions, such as "reviewing marketing strategies" and "optimizing advertising budgets," which is shared with meeting participants via their devices.

[0678] Examples of prompt statements

[0679] "Please display information about the target market for any new products discussed during the meeting. Also provide examples of past successful marketing strategies."

[0680] As described above, the present invention realizes appropriate information provision in real time and efficient conference follow-up.

[0681] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0682] Step 1:

[0683] The device captures video of the conference room with a camera and sends the data to the server in real time. This input data is a video stream, and each frame contains the faces of the conference participants. The server receives this data and performs preprocessing for face identification. Specifically, it analyzes the video data and detects and extracts faces from each frame.

[0684] Step 2:

[0685] The server applies a facial recognition algorithm (e.g., OpenCV or Dlib) to the preprocessed image data, extracts the features of the recognized face, and compares them with a pre-registered database. The matching results are output as the ID and name of the identified conference participant.

[0686] Step 3:

[0687] The server generates an attendee list based on the matching results. This list includes the name and related information of each conference participant. The generated attendee list is sent from the server to the terminal and displayed on the terminal's conference screen. This data output is displayed on the conference screen as text information.

[0688] Step 4:

[0689] The user speaks a question, and the device captures and sends the audio to the server. The input data is an audio file or a real-time audio stream. The server receives the data and performs preprocessing for speech recognition, such as noise reduction and speech normalization.

[0690] Step 5:

[0691] The server uses speech recognition technology (e.g., Google Speech-to-Text API) to convert the preprocessed voice data into text, and the output is the question in text format.

[0692] Step 6:

[0693] The server analyzes the text of the question using natural language processing (NLP) technology (e.g., SpaCy). As a result of this analysis, the intent of the question and related keywords are extracted.

[0694] Step 7:

[0695] The server searches the database for relevant materials and information based on the analysis results. This process uses a search engine such as Elasticsearch. The search results include links to related materials and summaries.

[0696] Step 8:

[0697] The server generates the necessary materials based on the search results and sends them to the terminal. The generated materials are displayed on the terminal's conference screen. This output data includes PDFs and presentation slides.

[0698] Step 9:

[0699] After the meeting ends, the server analyzes the accumulated meeting data (audio, text, and materials). It uses a natural language processing toolkit (e.g., NLTK, BERT) to extract and classify the issues discussed. The output is a list of the extracted issues and their categorization.

[0700] Step 10:

[0701] The server generates solutions to each problem based on the analysis results. Using a generative AI model (e.g., GPT-4), it outputs optimal solutions and predictions. These are sent to the terminal in the form of a report and shared with the meeting participants.

[0702] Step 11:

[0703] The device displays the received report to the meeting participants, allowing them to view the shared information, which allows for efficient follow-up after the meeting has ended.

[0704] In this way, the overall process of meeting progress and follow-up is detailed for each processing step.

[0705] (Application example 1)

[0706] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0707] Conventional conference systems have difficulty recognizing the faces of conference participants, processing questions in real time, generating materials instantly, and analyzing issues and presenting solutions after the conference has ended. Furthermore, in customer service at brick-and-mortar stores, there are challenges in effectively identifying customers' faces and providing personalized service, making it difficult to improve customer satisfaction. Furthermore, there is a need to analyze customer voices and instantly present in-store product information and inventory status. To solve these problems, a system capable of more efficient, real-time information processing and personalized response is needed.

[0708] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0709] In this invention, the server includes means for identifying the faces of conference participants, means for generating a list of identified conference participants, means for converting the content of questions from conference participants into text in real time using voice recognition technology, means for searching for and generating materials and information related to the content of the questions, means for categorizing issues discussed during the conference and presenting analysis results and solutions, means installed in smart glasses for identifying customers through entity recognition and providing personalized service, and means for analyzing the customer's voice and presenting product information and inventory status in the store. This enables appropriate information provision and efficient follow-up of issues in conferences, and enables real-time, personalized support to improve customer satisfaction in physical stores.

[0710] The "means for identifying the faces of conference participants" refers to a method of capturing the faces of conference participants using a camera and identifying individual participants using a facial recognition algorithm.

[0711] The "means for generating a list of identified conference participants" is a method for organizing and displaying information about conference participants identified by facial recognition in list form.

[0712] "Means for converting questions from conference participants into text in real time using speech recognition technology" refers to a method for converting speech spoken by conference participants into text data in real time using speech recognition technology.

[0713] "Means for searching and generating materials and information related to the question content" refers to a method for searching a database for related materials and information based on the analyzed question content, and generating and displaying them.

[0714] "Method of categorizing the issues discussed during the meeting and presenting the analysis results and solutions" is a method of organizing and analyzing the content discussed during the meeting by category, and presenting the analysis results and solutions based on that.

[0715] "Means installed on smart glasses for identifying customers through entity recognition and providing personalized service" refers to a method for identifying customers visiting a store using a camera and a recognition system installed in smart glasses and providing personalized service to the customer.

[0716] "Means for analyzing customer voice and presenting in-store product information and stock status" refers to a method for analyzing customer voice in real time and displaying and providing in-store product information and stock status based on the analysis results.

[0717] The present invention is a system including two main application examples: a conference system and a brick-and-mortar store system. This system is configured with a server, a terminal, and a user as its main components.

[0718] Conference System

[0719] server

[0720] The server has the following main functions:

[0721] 1. Facial Identification: Identify the faces of meeting participants from camera footage and compare them with a pre-registered database. OpenCV and TensorFlow are used for facial identification.

[0722] 2. Speech recognition: The user's question is converted into text using PyAudio and SpeechRecognition technology, and the content is analyzed.

[0723] 3. Document search: Search and generate documents and information related to the question from the database. This is done using SQLAlchemy.

[0724] 4. Issue analysis: Analyze the data accumulated during the meeting and classify and analyze the issues discussed by category.

[0725] 5. Solution Proposal: Based on the analysis results, provide future solutions and predicted results.

[0726] Terminal

[0727] The terminal has the following main features:

[0728] 1. Camera and microphone: Captures video and audio of meeting participants and sends them to the server.

[0729] 2. Display device: Displays materials and information sent from the server on the conference screen.

[0730] User

[0731] A user is an individual who participates in a conference and performs the following actions:

[0732] 1. Audio Questions: Ask questions by voice during the meeting.

[0733] 2. Viewing materials: View materials and information displayed on the terminal.

[0734] Physical store compatible system

[0735] Smart Glasses

[0736] The smart glasses have the following main features:

[0737] 1. Facial Recognition: Recognize the faces of customers who visit your physical store and identify repeat and new customers. OpenCV and TensorFlow are used for facial recognition.

[0738] 2. Speech Recognition: Analyze customer questions in real time and provide appropriate information, using PyAudio and SpeechRecognition technology.

[0739] 3. Product information display: Display product information and stock status based on customer questions. Use SQLAlchemy for database management.

[0740] server

[0741] The server has the following main functions:

[0742] 1. Facial recognition: Identify customers using video transmitted from smart glasses.

[0743] 2. Speech recognition: Converts the customer's voice into text and analyzes its content.

[0744] 3. Information search: Search and generate in-store product information and inventory status based on customer questions.

[0745] 4. History management: Maintain customer visit and purchase history to provide personalized service.

[0746] Specific examples

[0747] For example, if a repeat customer visits a store, the camera in the smart glasses will recognize the customer and respond with a voice message saying, "Welcome. Thank you for visiting again." If the customer asks, "Is this item in stock?", the voice recognition system will respond, "The item in question is currently in stock."

[0748] Also, if a participant asks during a meeting, "Is there a recent report on this issue?", the server will convert the speech into text, search for relevant materials, and display them.

[0749] Prompt Sentence Examples

[0750] "Generate program code for a smart glasses assistant that recognizes customer faces, greets returning and new customers appropriately, and provides inventory information and product recommendations based on customer questions. Use OpenCV and TensorFlow for face recognition, PyAudio and SpeechRecognition for voice recognition, and SQLAlchemy for database management."

[0751] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0752] Step 1:

[0753] Facial Recognition Processing

[0754] Server: Receives video footage of the conference room or store from the device and extracts facial features using OpenCV. Next, it uses TensorFlow's facial recognition algorithm to calculate facial features and compare them with facial data in the database.

[0755] Input: Video data from the device.

[0756] Output: A list of recognized meeting participants or customers.

[0757] Specific operation: The camera detects faces in each captured video frame, calculates features, and matches them with face information in a matching database.

[0758] Step 2:

[0759] Generate and display a list of conference participants

[0760] Server: Generates a list of identified conference participants based on facial recognition and sends the list to the terminal.

[0761] Input: Face identification results.

[0762] Output: A list of conference participants.

[0763] Specific operation: The recognized participant's name or ID is added to a list, sent to the device, and displayed.

[0764] Step 3:

[0765] Capture and transcribing voice questions

[0766] Terminal: The microphone captures the user's voice and sends it to the server.

[0767] Server: Converts received audio data into text using PyAudio and SpeechRecognition technology.

[0768] Input: Audio data from the user.

[0769] Output: Textualized question.

[0770] Specific operation: The user's voice is detected by the microphone, sent to the server in real time, and converted into text using voice recognition technology.

[0771] Step 4:

[0772] Searching for and generating materials and information

[0773] Server: Analyzes the textual content of the question, searches for and generates relevant materials and information from the database using SQLAlchemy, and sends the materials to the terminal.

[0774] Input: The text of the question.

[0775] Output: Related materials and information.

[0776] Specific operation: The database is searched using the question as a keyword, and relevant documents and information are extracted and sent to the terminal.

[0777] Step 5:

[0778] Categorizing problems and presenting solutions

[0779] Server: Analyzes questions and discussions accumulated during the meeting and categorizes them. Generates solutions for each category and sends detailed analysis results to the device.

[0780] Input: Questions and discussions during the meeting.

[0781] Output: Categorised challenges and their solutions.

[0782] Specific operation: The accumulated data is analyzed using natural language processing technology and classified into categories, and then a solution is generated using an AI model and sent to the terminal in report format.

[0783] Step 6:

[0784] Identifying and responding to customers in-store

[0785] Smart glasses: Capture the face of the customer and send it to the server. Based on the results of the server's matching, the glasses present the customer with appropriate greetings and information.

[0786] Input: Customer video data.

[0787] Output: Customer identification and response message.

[0788] How it works: The smart glasses' camera captures the customer's face and sends it to the server. When the recognition result is returned, a voice or visual response is given based on that data.

[0789] Step 7:

[0790] Analysis of customer voice questions and information presentation

[0791] Smart glasses: Capture customer voice questions with a microphone and send them to a server.

[0792] Server: Analyzes the voice data, searches for and generates product information and stock status based on the question, and sends that information to the smart glasses.

[0793] Input: Voice data from the customer.

[0794] Output: Product information and stock status.

[0795] Specific operation: The customer's voice is captured by a microphone and sent to a server, where it is analyzed using voice recognition technology, and the appropriate information is searched for in a database and sent to and displayed on the smart glasses.

[0796] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0797] This system combines an automatic announcer that recognizes the faces of conference participants and processes questions in real time with an emotion engine that recognizes the user's emotions. This system supports the smooth progress of conferences, provides information according to the participants' emotions, and efficiently follows up after the conference.

[0798] Overall system configuration

[0799] The system mainly includes three main components and one additional component: a server, a terminal, a user, and an emotion engine.

[0800] 1. Server

[0801] The server is the center of the system and has the following main functions:

[0802] Facial recognition: The faces of meeting participants are identified from camera footage and compared with a pre-registered database.

[0803] Question content analysis: The user's question is converted into text using voice recognition technology, and the content is analyzed.

[0804] Material search: Search and generate the necessary materials and information from the database.

[0805] Issue analysis: Analyze the data accumulated during the meeting and classify and analyze the issues discussed by category.

[0806] Solution Proposal: Based on the analysis results, we provide future solutions and predicted results.

[0807] Sentiment data analysis: Receive data from the sentiment engine and use it for follow-up during and after the meeting.

[0808] 2. Terminal

[0809] The terminal is an input device for in-room and remote participants and has the following main functions:

[0810] Camera and microphone: Captures video and audio of meeting participants and sends them to the server.

[0811] Display device: Displays materials and information sent from the server on the conference screen.

[0812] Emotion recognition: Analyze participants' facial expressions and tone of voice to obtain emotional data.

[0813] 3. Users

[0814] A user is an individual who participates in a conference and performs the following actions:

[0815] Voice questions: Ask voice questions during the meeting.

[0816] Viewing materials: View materials and information displayed on the device.

[0817] 4. Emotion Engine

[0818] The emotion engine is a component that recognizes and analyzes the user's emotions.

[0819] Emotion analysis: Analyze user emotions in real time through facial expression recognition and voice analysis.

[0820] Data provision: Analysis results are sent to the server and used to dynamically change materials and information.

[0821] Program processing explanation

[0822] The system's programs are processed in cooperation with the server, terminal, user, and emotion engine.

[0823] Facial recognition and participant list creation

[0824] 1. The device captures video from within the conference room using a camera and transmits it to the server in real time.

[0825] 2. The server applies a facial recognition algorithm to identify the attendee's face and match it with a database.

[0826] 3. The server generates an attendee list based on the matching results and sends it to the terminal.

[0827] 4. The terminal displays the list of conference participants on the screen.

[0828] Real-time question processing and answer generation

[0829] 1. The user speaks a question, and the device captures the audio and sends it to the server.

[0830] 2. The server uses a speech recognition engine to convert the voice data into text and analyze the content.

[0831] 3. The server searches and generates relevant materials and information from the database.

[0832] 4. The server sends the generated materials to the terminal, which displays them on the conference screen.

[0833] Sentiment analysis and information provision

[0834] 1. The device captures the user's facial expressions and tone of voice using an emotion engine and analyzes them in real time.

[0835] 2. The emotion engine sends the acquired emotion data to the server.

[0836] 3. The server dynamically changes the information provided based on the emotion data and updates materials and information as necessary.

[0837] Post-conference analysis and solution presentation

[0838] 1. The server analyzes the data (voice, text, materials, emotional data) accumulated during the meeting.

[0839] 2. The server uses this data to categorize the issues discussed and conduct a detailed analysis.

[0840] 3. The server compiles solutions and prediction results for each problem into a report, sends it to the terminal, and shares it with the meeting participants.

[0841] Specific examples

[0842] For example, consider a case where a company holds a meeting to discuss marketing strategies for a new product.

[0843] Step 1: When the meeting starts, the device captures the video of the meeting room, and the server identifies all attendees and displays the participant list.

[0844] Step 2: When a user asks, "What is the target market for the new product?" the device captures the voice, the server converts it into text, and generates and displays relevant marketing materials.

[0845] Step 3: During the meeting, if the emotion engine senses the user's doubts or concerns, the server updates the materials and provides additional information.

[0846] Step 4: After the meeting ends, the server analyzes the discussion content and emotional data, generates a report presenting issues and solutions such as "reviewing marketing strategies" and "optimizing advertising budgets," and shares this report with meeting participants via their devices.

[0847] In this way, the present invention makes it possible to provide appropriate information in real time, dynamically change materials according to the emotions of participants, and also realizes the analysis of problems and the presentation of solutions after the meeting has ended.

[0848] The processing flow will be explained below.

[0849] Step 1:

[0850] The device captures video data from within the conference room using a camera and transmits it to the server in real time.

[0851] Step 2:

[0852] The server receives the captured video data and applies a facial recognition algorithm to identify the faces of the meeting participants. Specifically, it uses a face detection model (e.g., Haar Cascade or MTCNN) to identify facial features, then extracts and matches them with face data in a database.

[0853] Step 3:

[0854] The server generates a list of identified conference participants based on the matching result and transmits the generated list to the terminal.

[0855] Step 4:

[0856] The device displays the list of conference participants received on the screen. For example, "Yamada Taro - Project Manager" or "Sato Hanako - Engineer."

[0857] Step 5:

[0858] The user speaks a question, for example, "What is the target market for the new product?"

[0859] Step 6:

[0860] The terminal captures the voice data of the question with a microphone device and transmits the captured voice data to the server.

[0861] Step 7:

[0862] The server converts the voice data into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[0863] Step 8:

[0864] The server analyzes the textual question using natural language processing (NLP) technology, for example, to extract keywords such as "target market" from the question.

[0865] Step 9:

[0866] The server searches the database and generates materials and information related to the query (for example, presentation slides on marketing strategies).

[0867] Step 10:

[0868] The server transmits the generated materials to the terminal, which then displays them on the conference screen.

[0869] Step 11:

[0870] The device captures the user's facial expressions and tone of voice using an emotion engine and analyzes them in real time.

[0871] Step 12:

[0872] The emotion engine sends the acquired emotion data to the server. For example, if the user feels anxious or suspicious about a question, that emotion is sent to the server.

[0873] Step 13:

[0874] The server dynamically changes the information provided based on the emotion data, and searches and generates additional materials and information from the database as needed.

[0875] Step 14:

[0876] The server generates additional materials and sends them to the terminal, which then displays them on the conference screen. For example, if the user is feeling anxious, detailed statistical data or additional explanatory materials can be displayed.

[0877] Step 15:

[0878] After the meeting ends, the server analyzes the data accumulated during the meeting (voice, text, materials, and emotional data).

[0879] Step 16:

[0880] Based on the analyzed data, the server categorizes the issues discussed, such as "reviewing marketing strategies" or "optimizing advertising budgets."

[0881] Step 17:

[0882] The server performs a detailed analysis of each category and considers solutions based on relevant data.

[0883] Step 18:

[0884] The server compiles solutions and prediction results for each problem into a report.

[0885] Step 19:

[0886] The server sends the generated report to the terminal and shares it with the conference participants through the terminal. For example, after the conference, a report on the proposal for "optimizing the advertising budget" and its feasibility is shared.

[0887] This series of steps allows for efficient management of the progress of meetings, provision of appropriate information in real time, and post-meeting analysis of issues and presentation of solutions.It also enables dynamic changes to materials based on the emotions of participants.

[0888] Example 2

[0889] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0890] Conventional conference systems were able to convert participants' questions into text using facial recognition and voice recognition, and to search for and generate materials. However, they lacked the means to analyze participants' emotions in real time and dynamically update materials and information based on that. Another issue was the inability to provide predictive results through data analysis after the meeting. This could result in meetings not progressing smoothly and participants' satisfaction declining.

[0891] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for identifying the faces of the conference participants, means for generating a list of the identified conference participants, means for converting the content of the questions of the conference participants into text in real time using voice recognition technology, means for searching for and generating materials and information related to the content of the questions, means for capturing and analyzing emotional data of the conference participants, means for dynamically changing the information provided based on the emotional data and updating the materials and information as necessary, and means for categorizing issues discussed during the conference and presenting analysis results and solutions. This makes it possible to identify the faces and content of the questions of the conference participants as well as dynamically provide information according to the emotions of the participants, thereby realizing smooth progress of the conference and follow-up.

[0892] "Conference participant" refers to an individual who attends a conference and is a user of the system.

[0893] "Means for identifying faces" refers to a device or process that uses a facial recognition algorithm to identify conference participants based on camera footage.

[0894] A "facial recognition algorithm" is a set of mathematical formulas and programs that use image processing technology to recognize and identify individual faces.

[0895] "Means for generating a list of conference participants" refers to a device or process that organizes information about identified conference participants and displays or records it in list form.

[0896] "Voice recognition technology" is a technology that analyzes human voice and converts the content into text data.

[0897] "Means for converting questions into text" refers to a device or process that uses speech recognition technology to convert voice questions from conference participants into text data in real time.

[0898] "Means for retrieving and generating materials and information" refers to a device or process that retrieves the required materials and information from a database and generates them in an appropriate format.

[0899] "Means for capturing and analyzing emotional data" refers to a device or process that uses a camera or microphone to capture the facial expressions and tone of voice of meeting participants and analyzes them using an emotion analysis algorithm.

[0900] "Means for dynamically changing information provision based on emotional data" refers to a device or process that updates or changes the materials or information provided in real time based on analyzed emotional data.

[0901] "Means for categorizing issues and presenting analysis results and solutions" refers to a device or process that analyzes data from the content discussed during a meeting, categorizes it, and then proposes the analysis results and solutions based on them.

[0902] "Means for generating prediction results" refers to a device or process that analyzes the data accumulated after the meeting and generates results to predict future developments.

[0903] MODE FOR CARRYING OUT THE INVENTION

[0904] This system combines an automatic announcer that recognizes the faces of conference participants and processes their questions in real time with an emotion engine that recognizes and analyzes user emotions. This system supports the smooth progress of conferences, provides information according to the participants' emotions, and efficiently follows up after the conference.

[0905] Overall system configuration

[0906] The system mainly includes three main components and one additional component: a server, a terminal, a user, and an emotion engine.

[0907] 1. Server

[0908] The server is the center of the system and has the following main functions:

[0909] Facial Identification: Uses OpenCV to identify the faces of meeting participants from camera footage and match them with a pre-registered database.

[0910] Question content analysis: The user's question is converted into text using voice recognition technology (Google Cloud Speech-to-Text), and the content is analyzed.

[0911] Material search: Search and generate the necessary materials and information from the database.

[0912] Issue analysis: Analyze the data accumulated during the meeting and classify and analyze the issues discussed by category.

[0913] Solution Proposal: Based on the analysis results, we provide future solutions and predicted results.

[0914] Sentiment data analysis: Receive data from the sentiment engine and use it for follow-up during and after the meeting.

[0915] 2. Terminal

[0916] The terminal is an input device for in-room and remote participants and has the following main functions:

[0917] Camera and microphone: Captures video and audio of meeting participants and sends them to the server.

[0918] Display device: Displays materials and information sent from the server on the conference screen.

[0919] Emotion recognition: Analyze participants' facial expressions and tone of voice to obtain emotional data.

[0920] 3. Users

[0921] A user is an individual who participates in a conference and performs the following actions:

[0922] Voice questions: Ask voice questions during the meeting.

[0923] Viewing materials: View materials and information displayed on the device.

[0924] 4. Emotion Engine

[0925] The emotion engine is a component that recognizes and analyzes the user's emotions.

[0926] Emotion Analysis: Uses Microsoft Azure Emotion API to analyze user emotions in real time through facial expression recognition and voice analysis.

[0927] Data provision: Analysis results are sent to the server and used to dynamically change materials and information.

[0928] Specific examples

[0929] For example, consider a case where a company holds a meeting to discuss marketing strategies for a new product.

[0930] Step 1: At the start of the meeting, the device captures the video of the meeting room, and the server identifies all attendees. A participant list is created and displayed on the device.

[0931] Step 2: When a user asks, "What is the target market for the new product?", the device captures the voice, the server converts the voice into text, and generates and displays related materials.

[0932] Step 3: During the meeting, if the emotion engine senses the user's doubts or concerns, the server updates the materials and provides additional information.

[0933] Step 4: After the meeting ends, the server analyzes the discussion content and emotional data, generates a report presenting issues and solutions such as "reviewing marketing strategies" and "optimizing advertising budgets," and shares this report with meeting participants via their devices.

[0934] These functions are important for ensuring smooth progress in meetings and providing appropriate information according to the emotions of participants.

[0935] Prompt Sentence Examples

[0936] Describe how your system will respond if a user asks, "What is the target market for the new product?"

[0937] As described above, the present invention makes it possible to provide appropriate information in real time, dynamically change materials in response to the emotions of participants, and also to analyze issues and present solutions after the meeting has ended.

[0938] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0939] Step 1: Capture footage

[0940] Input: Camera video sent from the device.

[0941] Processing: The device captures video in real time through the camera in the conference room and sends the data to the server.

[0942] Output: Captured video data.

[0943] Step 2: Facial Recognition and Matching

[0944] Input: Captured video data.

[0945] Processing: The server runs a facial recognition algorithm using OpenCV to identify participants' faces from the video data and match them with pre-registered facial data in a database.

[0946] Output: A list of identified conference participants.

[0947] Step 3: Generate a participant list

[0948] Input: A list of identified conference participants.

[0949] Processing: The server generates this list and sends it to the terminal.

[0950] Output: Participant list sent to the device.

[0951] Step 4: View the Participant List

[0952] Input: The participant list sent to the device.

[0953] Processing: The terminal displays the received participant list on the conference screen. For example, names such as "Taro Tanaka" and "Hanako Sato" are displayed in the list.

[0954] Output: Participant list displayed on the conference screen.

[0955] Step 5: Capture audio

[0956] Input: User's spoken question.

[0957] Processing: During a meeting, a user asks a question by voice, such as "Where are the marketing materials?" The device captures the voice with its microphone and sends it to the server as audio data.

[0958] Output: The audio data sent to the server.

[0959] Step 6: Speech to Text

[0960] Input: The audio data sent to the server.

[0961] Processing: The server uses Google Cloud Speech-to-Text to convert the audio data into text. For example, the audio is converted into text, such as "Where are the marketing materials?"

[0962] Output: The question converted to text.

[0963] Step 7: Parsing the Question

[0964] Input: The question converted to text.

[0965] Processing: The server analyzes the textual question and searches the database for relevant information and materials.

[0966] Output: Search results for related materials.

[0967] Step 8: Find and generate relevant materials

[0968] Input: Search results for related materials.

[0969] Processing: The server generates the necessary materials based on the search results, for example preparing a PDF file of marketing materials.

[0970] Output: The generated material.

[0971] Step 9: Viewing the material

[0972] Input: Generated materials.

[0973] Processing: The server sends the generated materials to the terminal, which displays them on the conference screen. The user can view the displayed materials.

[0974] Output: Materials displayed on the meeting screen.

[0975] Step 10: Capturing Emotional Data

[0976] Input: Facial expressions and tone of voice of meeting participants.

[0977] Processing: The device uses a camera and microphone to capture emotional data such as the user's facial expressions and tone of voice in real time.

[0978] Output: Captured emotion data.

[0979] Step 11: Analyze the emotion data

[0980] Input: Captured emotion data.

[0981] Processing: An emotion engine (e.g., Microsoft Azure Emotion API) analyzes the captured data and determines the user's emotion. For example, it outputs an analysis result such as "The user looks anxious."

[0982] Output: Parsed emotion data.

[0983] Step 12: Dynamic information provision and documentation updates

[0984] Input: Parsed emotion data.

[0985] Processing: The emotion engine sends the analysis results to the server. The server then provides dynamic information based on the emotion data, adapting to the situation. For example, if the user appears anxious, the server provides additional materials with additional explanations. The device then displays the latest materials on the conference screen.

[0986] Output: Updated documentation and information.

[0987] Step 13: Data collection and analysis

[0988] Input: Audio, text, materials, emotional data, etc. accumulated during the meeting.

[0989] Processing: The server centralizes this data and performs detailed analysis.

[0990] Output: Analysis results.

[0991] Step 14: Categorize and analyze the issue

[0992] Input: Analysis results.

[0993] Processing: Based on the analysis results, the server categorizes the issues discussed and conducts a detailed analysis.

[0994] Output: A list of issues sorted by category.

[0995] Step 15: Generate and share your report

[0996] Input: A list of issues sorted by category.

[0997] Processing: The server generates a report that describes the solutions and prediction results for each problem. The report is sent to the terminal and can be viewed by the meeting participants.

[0998] Output: Report shared with meeting participants.

[0999] (Application example 2)

[1000] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1001] Modern factories are required to immediately solve problems faced by workers and improve productivity. However, conventional systems lack real-time problem-solving capabilities and support that adapts to the emotions of workers. As a result, work efficiency declines and work errors become more likely to occur. The present invention aims to provide a real-time problem-solving assistant system specialized for factory environments to solve these problems.

[1002] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for identifying the faces of factory workers, means for generating a list of identified factory workers, means for converting the contents of questions from factory workers into text in real time using voice recognition technology, means for searching and generating procedures and information related to the contents of the questions, and means for classifying problems that occur during factory work into categories and presenting analysis results and solutions. This makes it possible to quickly solve problems faced by factory workers and improve work efficiency.

[1003] "Conference participant" refers to any person participating in a conference.

[1004] "Means for identifying faces" refers to technology that uses a camera or facial recognition algorithm to recognize the face of a target person and identify that person.

[1005] The "means for generating a list of identified conference participants" is a technology for creating a list of participants based on information about conference participants obtained by face recognition.

[1006] "Speech recognition technology" is a technology that converts human speech into text data in real time.

[1007] "Means for searching and generating materials and information" refers to technology that searches for the necessary materials and information from related databases and information sources based on the content of a user's question and provides it.

[1008] "Means of categorizing issues and presenting analysis results and solutions" is a technique for organizing and classifying problems and issues discussed during meetings or work, and proposing solutions to them.

[1009] "Factory worker" refers to a worker who is in charge of work in a factory.

[1010] "Means for identifying the faces of factory workers" refers to technology that uses cameras and facial recognition algorithms to recognize the faces of target workers and identify them.

[1011] The "means for generating a list of identified factory workers" is a technology for creating a list of on-site workers based on information about factory workers obtained through facial recognition.

[1012] "Means for converting the content of questions posed by factory workers into text in real time using voice recognition technology" refers to technology for converting the voices spoken by factory workers into text in real time.

[1013] The "means for searching and generating procedures and information related to the content of a question" is a technology that searches a database for and provides relevant work procedures and information based on the content of a question posed by a factory worker.

[1014] "Means of categorizing problems that occur during factory work and presenting analysis results and solutions" is a technique for organizing and classifying troubles and problems that occur during work and proposing solutions to them.

[1015] This invention is a real-time problem-solving assistant system for improving work efficiency in a factory environment. This system mainly consists of a server, a terminal, a user, and an emotion engine.

[1016] Server Features

[1017] The server is the center of the system and has the following important functions:

[1018] Facial recognition: Identify the faces of factory workers from camera footage and compare them with a pre-registered database. Uses the dlib library and OpenCV as facial recognition algorithms.

[1019] Question content analysis: The questions from the workers are converted into text using speech recognition technology, and the content is analyzed. This is done using the speech_recognition library.

[1020] Document search: Search and generate the necessary procedures and information from a database that stores work procedures and equipment information within the factory.

[1021] Problem analysis: Analyze the data accumulated during the work and classify and analyze the problems that arise by category.

[1022] Solution Proposal: Based on the analysis results, we provide future solutions and predicted results.

[1023] Emotion data analysis: Receive data from the emotion engine and use it for follow-up during and after work.

[1024] Device Features

[1025] The terminal is an input device used by factory workers and has the following functions:

[1026] Camera and microphone: Captures video and audio of factory workers and sends them to a server.

[1027] Display device: The procedure and information sent from the server are displayed on the work screen. Display devices include smart glasses and smartphones.

[1028] Emotion Recognition: Analyzes the facial expressions and tone of voice of workers to obtain emotional data. Emotion recognition is performed using the Affectiva API.

[1029] User Actions

[1030] The user refers to a factory worker and performs the following actions:

[1031] Voice questions: Ask questions by voice as you work.

[1032] View Instructions: View instructions and information displayed on your device.

[1033] Emotion Engine Functions

[1034] The emotion engine is a component that recognizes and analyzes the emotions of workers.

[1035] Emotion analysis: Analyze workers' emotions in real time through facial expression recognition and voice analysis.

[1036] Data provision: The analysis results are sent to the server and used to dynamically change procedures and information.

[1037] Specific examples

[1038] For example, consider a situation where a problem occurs during the operation of a machine in a factory.

[1039] Step 1: When work begins, the device captures the camera image, and the server identifies the worker. A list of workers is displayed.

[1040] Step 2: When the worker asks, "Why is the machine stuck?" the device captures the voice, the server transcribes it, and generates the relevant repair instructions.

[1041] Step 3: If the emotion engine senses confusion or anxiety from the worker during the work, the server updates the displayed repair procedure with additional information to support the worker.

[1042] Step 4: After the work is completed, the server analyzes the accumulated data and generates a report that presents solutions such as "machine maintenance methods" and "future improvements," which is shared with the worker via their terminal.

[1043] Prompt Sentence Examples

[1044] "Devant a system that describes the situation when a maintenance issue occurs in the factory and generates real-time solutions to specific questions. Also include the ability to understand worker sentiment and provide follow-up accordingly."

[1045] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1046] Step 1:

[1047] Input: A factory worker activates smart glasses and camera footage is captured.

[1048] Processing (data processing and calculation): The device (smart glasses) captures the video of the factory worker and sends it to the server. The server uses OpenCV and dlib libraries to perform facial recognition and compares it with existing data in the database.

[1049] Output: A list of identified factory workers is generated and sent to the terminal.

[1050] How it works: The server applies a facial recognition algorithm to identify factory workers, creates a list, and displays it on the device.

[1051] Step 2:

[1052] Input: A factory worker asks a voiceover, "Why is the machine stuck?"

[1053] Processing (data processing and calculation): The device captures the voice and sends it to the server. The server converts the voice data into text using the speech_recognition library and analyzes the question.

[1054] Output: Repair procedures and information related to the query are searched for and generated from the database.

[1055] Specific operation: The server searches the database based on the text of the question, and sends relevant repair procedures and information to the terminal, where they are displayed.

[1056] Step 3:

[1057] Input: As the factory worker continues to follow the displayed steps, the emotion engine captures their facial expressions and tone of voice.

[1058] Processing (data processing and calculation): The device captures emotion data and sends it to the emotion engine in real time. The emotion engine analyzes emotions using the Affectiva API.

[1059] Output: The captured emotion data is sent to a server, which provides additional information or assistance if needed.

[1060] Specific operation: The server receives the sentiment analysis results and provides further support information if the worker is experiencing difficulties.

[1061] Step 4:

[1062] Input: The worker completes the repair procedure and the data on the work is sent from the terminal to the server.

[1063] Processing (data processing and calculation): The server analyzes the data (voice, text, emotion data) accumulated during the work and categorizes the problems that arise. Based on the analysis results, it generates solutions and predictions.

[1064] Output: A report containing the analysis results and solutions is generated and sent to the device.

[1065] Specific operation: The server generates a report and sends it to the terminal to be shared with workers and managers.

[1066] Step 5:

[1067] Input: Server-generated reports are reviewed by operators and administrators.

[1068] Processing (data processing and calculation): The report is displayed on the terminal, and suggestions for improvements and maintenance methods for the next work are made as needed.

[1069] Output: Next work plan and improvement suggestions are provided as reference for workers and managers.

[1070] Specific action: Workers and managers use the report to plan the next work.

[1071] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1072] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1073] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1074] [Third embodiment]

[1075] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1076] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1077] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1078] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1079] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1080] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1081] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1082] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1083] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1084] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1085] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1086] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1087] This invention provides an automatic conference announcer system that recognizes the faces of conference participants and processes questions in real time. This system manages the conference from start to finish, instantly generating and providing necessary materials and information, and also analyzes issues and presents solutions after the conference ends.

[1088] Overall system configuration

[1089] The system comprises three main components: a server, a terminal, and a user.

[1090] 1. Server

[1091] The server is the heart of the system and has the following main functions:

[1092] Facial recognition: The faces of meeting participants are identified from camera footage and compared with a pre-registered database.

[1093] Question content analysis: The user's question is converted into text using voice recognition technology, and the content is analyzed.

[1094] Material search: Search and generate the necessary materials and information from the database.

[1095] Issue analysis: Analyze the data accumulated during the meeting and classify and analyze the issues discussed by category.

[1096] Solution Proposal: Based on the analysis results, we provide future solutions and predicted results.

[1097] 2. Terminal

[1098] The terminal is an input device for in-room and remote participants and has the following main functions:

[1099] Camera and microphone: Captures video and audio of meeting participants and sends them to the server.

[1100] Display device: Displays materials and information sent from the server on the conference screen.

[1101] 3. Users

[1102] A user is an individual who participates in a conference and performs the following actions:

[1103] Voice questions: Ask voice questions during the meeting.

[1104] Viewing materials: View materials and information displayed on the device.

[1105] Program processing explanation

[1106] The processing of the programs in this system is mainly carried out in cooperation between the server, the terminal, and the user.

[1107] Facial recognition and participant list creation

[1108] 1. The device captures video of the conference room with its camera and sends it to the server.

[1109] 2. The server uses a facial recognition algorithm to identify attendees' faces and compare them with a registered database. Based on the results of the comparison, it generates an attendee list, which is sent to the device and displayed on the screen.

[1110] Real-time question processing and answer generation

[1111] 1. The user speaks a question, and the device captures the audio and sends it to the server.

[1112] 2. The server uses voice recognition technology to convert the question into text and analyzes the content.

[1113] 3. The server searches the database for materials and information related to the question, generates the necessary materials, and sends them to the terminal to display on the screen.

[1114] Post-conference analysis and solution presentation

[1115] 1. The server analyzes the data (audio, text, materials) accumulated during the meeting and extracts the content discussed.

[1116] 2. The server classifies the extracted issues into categories and performs detailed analysis.

[1117] 3. The server generates solutions and prediction results for each problem, compiles them into a report, and sends it to the terminal to share with the meeting participants.

[1118] Specific examples

[1119] For example, consider a case where a company holds a meeting to discuss marketing strategies for a new product.

[1120] Step 1: When the meeting starts, the device captures the video of the meeting room, and the server identifies all attendees and displays a list of participants.

[1121] Step 2: When a user asks, "What is the target market for the new product?" the device captures the voice, the server converts it into text, and generates and displays relevant marketing materials.

[1122] Step 3: After the meeting, the server analyzes the content discussed and generates a report presenting issues and solutions, such as "reviewing marketing strategies" and "optimizing advertising budgets," which is shared with meeting participants via their devices.

[1123] In this way, the present invention allows for the provision of appropriate information in real time and efficient meeting follow-up.

[1124] The processing flow will be explained below.

[1125] Step 1:

[1126] The device captures video data from within the conference room using a camera and transmits it to the server in real time.

[1127] Step 2:

[1128] The server receives the captured video data and applies a facial recognition algorithm to identify the faces of the meeting participants. Specifically, it uses a face detection model (e.g., Haar Cascade or MTCNN) to identify facial features, then extracts and matches them with face data in a database.

[1129] Step 3:

[1130] The server generates a list of identified conference participants based on the matching result and transmits the generated list to the terminal.

[1131] Step 4:

[1132] The device displays the list of conference participants received on the screen. For example, "Yamada Taro - Project Manager" or "Sato Hanako - Engineer."

[1133] Step 5:

[1134] The user speaks a question, for example, "What is the target market for the new product?"

[1135] Step 6:

[1136] The terminal captures the voice data of the question with a microphone device and transmits the captured voice data to the server.

[1137] Step 7:

[1138] The server converts the voice data into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[1139] Step 8:

[1140] The server analyzes the textual question using natural language processing (NLP) technology, for example, to extract keywords such as "target market" from the question.

[1141] Step 9:

[1142] The server searches the database and generates materials and information related to the query (for example, presentation slides on marketing strategies).

[1143] Step 10:

[1144] The server transmits the generated materials to the terminal, which then displays them on the conference screen.

[1145] Step 11:

[1146] After the meeting ends, the server analyzes the data (audio, text, and materials) accumulated during the meeting.

[1147] Step 12:

[1148] Based on the analyzed data, the server categorizes the issues discussed, such as "reviewing marketing strategies" or "optimizing advertising budgets."

[1149] Step 13:

[1150] The server performs a detailed analysis of each category and considers solutions based on relevant data.

[1151] Step 14:

[1152] The server compiles solutions and prediction results for each problem into a report.

[1153] Step 15:

[1154] The server sends the generated report to the terminal and shares it with the conference participants through the terminal.

[1155] This series of steps enables efficient management of the meeting progress, provision of appropriate information in real time, and post-meeting analysis of issues and presentation of solutions.

[1156] Example 1

[1157] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1158] During a meeting, it is necessary to identify participants' faces, process questions in real time, search for and generate relevant materials, analyze the issues discussed, and present solutions. However, it is difficult to perform these tasks efficiently and automatically. It is also equally difficult to quickly analyze the content discussed after the meeting and present solutions afterward. Conventional systems only automate part of these processes, resulting in a decrease in overall efficiency.

[1159] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1160] In this invention, the server includes means for identifying the faces of conference participants, means for generating a list of identified conference participants, means for converting the contents of questions asked by the conference participants into text in real time using voice recognition technology, means for searching for and generating materials and information related to the contents of the questions, means for categorizing issues discussed during the conference and presenting analysis results and solutions, means for transmitting the generated materials and information to terminals and displaying them on the conference screen, and means for analyzing accumulated data after the conference ends, extracting and categorizing the contents discussed, and generating solutions. This makes it possible to automate and efficiently manage the overall process of conference progress and follow-up.

[1161] A "conference participant" is an individual participating in a conference.

[1162] A "means for identifying faces" is a technology or device for analyzing camera footage and recognizing the faces of individual conference participants.

[1163] The "means for generating a list of conference participants" is a technology or device that creates a list of people participating in a conference based on the results of face identification.

[1164] "Speech recognition technology" is a technology that converts voice data into text data in real time.

[1165] "Means for converting questions into text in real time using voice recognition technology" refers to a technology or device that captures questions uttered by conference participants as voice data and instantly converts it into text.

[1166] "Means for searching and generating materials and information" refers to technology or equipment that searches for necessary materials and information from a database and generates them on request.

[1167] A "means for categorizing issues" is a technique or device that separates the content discussed during a meeting into multiple categories.

[1168] "Means for presenting analysis results and solutions" refers to a technique or device that analyzes the content of the discussion and derives and presents solutions based on the results.

[1169] The "means for transmitting generated materials and information to the terminal" refers to a technology or device for transmitting various materials and information generated by the server to the terminal in the conference room.

[1170] The "means for displaying on the conference screen" refers to a technique or device for displaying materials and information sent to the terminal on the conference screen.

[1171] The "means for analyzing data accumulated after the end of a meeting" refers to a technology or device for analyzing data such as audio, text, and materials collected after the end of a meeting.

[1172] "Means for extracting and classifying the content discussed and generating solutions" refers to a technology or device that analyzes the content discussed during a meeting, extracts important elements, classifies them by category, and generates solutions.

[1173] This invention provides an automatic conference announcer system that recognizes the faces of conference participants and processes questions in real time. This system manages the conference from start to finish, instantly generating and providing necessary materials and information, and also analyzes issues and presents solutions after the conference ends.

[1174] Overall system configuration

[1175] The system comprises three main components: a server, a terminal, and a user.

[1176] 1. Server

[1177] The server is the heart of the system and has the following main functions:

[1178] Facial recognition: The server receives the video footage from the conference room camera sent from the device, identifies attendees using a facial recognition algorithm, and compares them with a registered database. Technologies used include OpenCV and Dlib.

[1179] Question analysis: The server receives the data captured by the device and sends the voice question from the user, and converts the question into text using speech recognition technology. This can be done using the Google Speech-to-Text API. The text question is then analyzed using natural language processing technology (e.g., SpaCy).

[1180] Material search: The server uses a search engine such as Elasticsearch to search and generate the necessary materials and information from the database.

[1181] Issue analysis: The server analyzes the data accumulated during the meeting and categorizes and analyzes the issues discussed using natural language processing toolkits (e.g., NLTK, BERT).

[1182] Solution Proposal: The server generates future solutions and predictions based on the analysis results, using a generative AI model (e.g., GPT-4) to provide the optimal solution.

[1183] 2. Terminal

[1184] The terminal is an input device for in-room and remote participants and has the following main functions:

[1185] Camera and microphone: The device captures the video and audio of meeting participants in real time and sends them to the server.

[1186] Display device: The terminal receives materials and information sent from the server and displays them on the conference screen. This can be an electronic whiteboard or video conferencing software (e.g., Zoom, Microsoft Teams).

[1187] 3. Users

[1188] A user is an individual who participates in a conference and performs the following actions:

[1189] Voice questions: Users ask questions by voice during the conference, which are captured by the terminal and sent to the server.

[1190] Viewing materials: Users view materials and information displayed on their terminals and obtain appropriate information as the meeting progresses.

[1191] Program processing explanation

[1192] Facial recognition and participant list creation

[1193] The device captures video of the conference room with its camera and sends it to the server, where the server uses a facial recognition algorithm to identify attendees' faces and compare them with a registered database. Based on the results of the comparison, a list of attendees is generated and sent to the device, where it is displayed on the screen.

[1194] Real-time question processing and answer generation

[1195] The user speaks a question, and the device captures the voice and sends it to the server. The server uses voice recognition technology to convert the question into text and analyzes it. The server then searches a database for materials and information related to the question and generates the necessary materials. These are then sent to the device and displayed on the screen.

[1196] Post-conference analysis and solution presentation

[1197] The server analyzes the data (audio, text, and materials) accumulated during the meeting and extracts the content of the discussion. The server then categorizes the extracted issues and performs a detailed analysis. The server then generates solutions and predictions for each issue, compiles them into a report, and sends it to the device to share with the meeting participants.

[1198] Specific examples

[1199] For example, consider a case where a company holds a meeting to discuss marketing strategies for a new product.

[1200] Step 1: When the meeting starts, the device captures the video of the meeting room, and the server identifies all attendees and displays a list of participants.

[1201] Step 2: When a user asks, "What is the target market for the new product?" the device captures the voice, the server converts it into text, and generates and displays relevant marketing materials.

[1202] Step 3: After the meeting, the server analyzes the content discussed and generates a report presenting issues and solutions, such as "reviewing marketing strategies" and "optimizing advertising budgets," which is shared with meeting participants via their devices.

[1203] Examples of prompt statements

[1204] "Please display information about the target market for any new products discussed during the meeting. Also provide examples of past successful marketing strategies."

[1205] As described above, the present invention realizes appropriate information provision in real time and efficient conference follow-up.

[1206] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1207] Step 1:

[1208] The device captures video of the conference room with a camera and sends the data to the server in real time. This input data is a video stream, and each frame contains the faces of the conference participants. The server receives this data and performs preprocessing for face identification. Specifically, it analyzes the video data and detects and extracts faces from each frame.

[1209] Step 2:

[1210] The server applies a facial recognition algorithm (e.g., OpenCV or Dlib) to the preprocessed image data, extracts the features of the recognized face, and compares them with a pre-registered database. The matching results are output as the ID and name of the identified conference participant.

[1211] Step 3:

[1212] The server generates an attendee list based on the matching results. This list includes the name and related information of each conference participant. The generated attendee list is sent from the server to the terminal and displayed on the terminal's conference screen. This data output is displayed on the conference screen as text information.

[1213] Step 4:

[1214] The user speaks a question, and the device captures and sends the audio to the server. The input data is an audio file or a real-time audio stream. The server receives the data and performs preprocessing for speech recognition, such as noise reduction and speech normalization.

[1215] Step 5:

[1216] The server uses speech recognition technology (e.g., Google Speech-to-Text API) to convert the preprocessed voice data into text, and the output is the question in text format.

[1217] Step 6:

[1218] The server analyzes the text of the question using natural language processing (NLP) technology (e.g., SpaCy). As a result of this analysis, the intent of the question and related keywords are extracted.

[1219] Step 7:

[1220] The server searches the database for relevant materials and information based on the analysis results. This process uses a search engine such as Elasticsearch. The search results include links to related materials and summaries.

[1221] Step 8:

[1222] The server generates the necessary materials based on the search results and sends them to the terminal. The generated materials are displayed on the terminal's conference screen. This output data includes PDFs and presentation slides.

[1223] Step 9:

[1224] After the meeting ends, the server analyzes the accumulated meeting data (audio, text, and materials). It uses a natural language processing toolkit (e.g., NLTK, BERT) to extract and classify the issues discussed. The output is a list of the extracted issues and their categorization.

[1225] Step 10:

[1226] The server generates solutions to each problem based on the analysis results. Using a generative AI model (e.g., GPT-4), it outputs optimal solutions and predictions. These are sent to the terminal in the form of a report and shared with the meeting participants.

[1227] Step 11:

[1228] The device displays the received report to the meeting participants, allowing them to view the shared information, which allows for efficient follow-up after the meeting has ended.

[1229] In this way, the overall process of meeting progress and follow-up is detailed for each processing step.

[1230] (Application example 1)

[1231] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1232] Conventional conference systems have difficulty recognizing the faces of conference participants, processing questions in real time, generating materials instantly, and analyzing issues and presenting solutions after the conference has ended. Furthermore, in customer service at brick-and-mortar stores, there are challenges in effectively identifying customers' faces and providing personalized service, making it difficult to improve customer satisfaction. Furthermore, there is a need to analyze customer voices and instantly present in-store product information and inventory status. To solve these problems, a system capable of more efficient, real-time information processing and personalized response is needed.

[1233] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1234] In this invention, the server includes means for identifying the faces of conference participants, means for generating a list of identified conference participants, means for converting the content of questions from conference participants into text in real time using voice recognition technology, means for searching for and generating materials and information related to the content of the questions, means for categorizing issues discussed during the conference and presenting analysis results and solutions, means installed in smart glasses for identifying customers through entity recognition and providing personalized service, and means for analyzing the customer's voice and presenting product information and inventory status in the store. This enables appropriate information provision and efficient follow-up of issues in conferences, and enables real-time, personalized support to improve customer satisfaction in physical stores.

[1235] The "means for identifying the faces of conference participants" refers to a method of capturing the faces of conference participants using a camera and identifying individual participants using a facial recognition algorithm.

[1236] The "means for generating a list of identified conference participants" is a method for organizing and displaying information about conference participants identified by facial recognition in list form.

[1237] "Means for converting questions from conference participants into text in real time using speech recognition technology" refers to a method for converting speech spoken by conference participants into text data in real time using speech recognition technology.

[1238] "Means for searching and generating materials and information related to the question content" refers to a method for searching a database for related materials and information based on the analyzed question content, and generating and displaying them.

[1239] "Method of categorizing the issues discussed during the meeting and presenting the analysis results and solutions" is a method of organizing and analyzing the content discussed during the meeting by category, and presenting the analysis results and solutions based on that.

[1240] "Means installed on smart glasses for identifying customers through entity recognition and providing personalized service" refers to a method for identifying customers visiting a store using a camera and a recognition system installed in smart glasses and providing personalized service to the customer.

[1241] "Means for analyzing customer voice and presenting in-store product information and stock status" refers to a method for analyzing customer voice in real time and displaying and providing in-store product information and stock status based on the analysis results.

[1242] The present invention is a system including two main application examples: a conference system and a brick-and-mortar store system. This system is configured with a server, a terminal, and a user as its main components.

[1243] Conference System

[1244] server

[1245] The server has the following main functions:

[1246] 1. Facial Identification: Identify the faces of meeting participants from camera footage and compare them with a pre-registered database. OpenCV and TensorFlow are used for facial identification.

[1247] 2. Speech recognition: The user's question is converted into text using PyAudio and SpeechRecognition technology, and the content is analyzed.

[1248] 3. Document search: Search and generate documents and information related to the question from the database. This is done using SQLAlchemy.

[1249] 4. Issue analysis: Analyze the data accumulated during the meeting and classify and analyze the issues discussed by category.

[1250] 5. Solution Proposal: Based on the analysis results, provide future solutions and predicted results.

[1251] Terminal

[1252] The terminal has the following main features:

[1253] 1. Camera and microphone: Captures video and audio of meeting participants and sends them to the server.

[1254] 2. Display device: Displays materials and information sent from the server on the conference screen.

[1255] User

[1256] A user is an individual who participates in a conference and performs the following actions:

[1257] 1. Audio Questions: Ask questions by voice during the meeting.

[1258] 2. Viewing materials: View materials and information displayed on the terminal.

[1259] Physical store compatible system

[1260] Smart Glasses

[1261] The smart glasses have the following main features:

[1262] 1. Facial Recognition: Recognize the faces of customers who visit your physical store and identify repeat and new customers. OpenCV and TensorFlow are used for facial recognition.

[1263] 2. Speech Recognition: Analyze customer questions in real time and provide appropriate information, using PyAudio and SpeechRecognition technology.

[1264] 3. Product information display: Display product information and stock status based on customer questions. Use SQLAlchemy for database management.

[1265] server

[1266] The server has the following main functions:

[1267] 1. Facial recognition: Identify customers using video transmitted from smart glasses.

[1268] 2. Speech recognition: Converts the customer's voice into text and analyzes its content.

[1269] 3. Information search: Search and generate in-store product information and inventory status based on customer questions.

[1270] 4. History management: Maintain customer visit and purchase history to provide personalized service.

[1271] Specific examples

[1272] For example, if a repeat customer visits a store, the camera in the smart glasses will recognize the customer and respond with a voice message saying, "Welcome. Thank you for visiting again." If the customer asks, "Is this item in stock?", the voice recognition system will respond, "The item in question is currently in stock."

[1273] Also, if a participant asks during a meeting, "Is there a recent report on this issue?", the server will convert the speech into text, search for relevant materials, and display them.

[1274] Prompt Sentence Examples

[1275] "Generate program code for a smart glasses assistant that recognizes customer faces, greets returning and new customers appropriately, and provides inventory information and product recommendations based on customer questions. Use OpenCV and TensorFlow for face recognition, PyAudio and SpeechRecognition for voice recognition, and SQLAlchemy for database management."

[1276] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1277] Step 1:

[1278] Facial Recognition Processing

[1279] Server: Receives video footage of the conference room or store from the device and extracts facial features using OpenCV. Next, it uses TensorFlow's facial recognition algorithm to calculate facial features and compare them with facial data in the database.

[1280] Input: Video data from the device.

[1281] Output: A list of recognized meeting participants or customers.

[1282] Specific operation: The camera detects faces in each captured video frame, calculates features, and matches them with face information in a matching database.

[1283] Step 2:

[1284] Generate and display a list of conference participants

[1285] Server: Generates a list of identified conference participants based on facial recognition and sends the list to the terminal.

[1286] Input: Face identification results.

[1287] Output: A list of conference participants.

[1288] Specific operation: The recognized participant's name or ID is added to a list, sent to the device, and displayed.

[1289] Step 3:

[1290] Capture and transcribing voice questions

[1291] Terminal: The microphone captures the user's voice and sends it to the server.

[1292] Server: Converts received audio data into text using PyAudio and SpeechRecognition technology.

[1293] Input: Audio data from the user.

[1294] Output: Textualized question.

[1295] Specific operation: The user's voice is detected by the microphone, sent to the server in real time, and converted into text using voice recognition technology.

[1296] Step 4:

[1297] Searching for and generating materials and information

[1298] Server: Analyzes the textual content of the question, searches for and generates relevant materials and information from the database using SQLAlchemy, and sends the materials to the terminal.

[1299] Input: The text of the question.

[1300] Output: Related materials and information.

[1301] Specific operation: The database is searched using the question as a keyword, and relevant documents and information are extracted and sent to the terminal.

[1302] Step 5:

[1303] Categorizing problems and presenting solutions

[1304] Server: Analyzes questions and discussions accumulated during the meeting and categorizes them. Generates solutions for each category and sends detailed analysis results to the device.

[1305] Input: Questions and discussions during the meeting.

[1306] Output: Categorised challenges and their solutions.

[1307] Specific operation: The accumulated data is analyzed using natural language processing technology and classified into categories, and then a solution is generated using an AI model and sent to the terminal in report format.

[1308] Step 6:

[1309] Identifying and responding to customers in-store

[1310] Smart glasses: Capture the face of the customer and send it to the server. Based on the results of the server's matching, the glasses present the customer with appropriate greetings and information.

[1311] Input: Customer video data.

[1312] Output: Customer identification and response message.

[1313] How it works: The smart glasses' camera captures the customer's face and sends it to the server. When the recognition result is returned, a voice or visual response is given based on that data.

[1314] Step 7:

[1315] Analysis of customer voice questions and information presentation

[1316] Smart glasses: Capture customer voice questions with a microphone and send them to a server.

[1317] Server: Analyzes the voice data, searches for and generates product information and stock status based on the question, and sends that information to the smart glasses.

[1318] Input: Voice data from the customer.

[1319] Output: Product information and stock status.

[1320] Specific operation: The customer's voice is captured by a microphone and sent to a server, where it is analyzed using voice recognition technology, and the appropriate information is searched for in a database and sent to and displayed on the smart glasses.

[1321] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1322] This system combines an automatic announcer that recognizes the faces of conference participants and processes questions in real time with an emotion engine that recognizes the user's emotions. This system supports the smooth progress of conferences, provides information according to the participants' emotions, and efficiently follows up after the conference.

[1323] Overall system configuration

[1324] The system mainly includes three main components and one additional component: a server, a terminal, a user, and an emotion engine.

[1325] 1. Server

[1326] The server is the center of the system and has the following main functions:

[1327] Facial recognition: The faces of meeting participants are identified from camera footage and compared with a pre-registered database.

[1328] Question content analysis: The user's question is converted into text using voice recognition technology, and the content is analyzed.

[1329] Material search: Search and generate the necessary materials and information from the database.

[1330] Issue analysis: Analyze the data accumulated during the meeting and classify and analyze the issues discussed by category.

[1331] Solution Proposal: Based on the analysis results, we provide future solutions and predicted results.

[1332] Sentiment data analysis: Receive data from the sentiment engine and use it for follow-up during and after the meeting.

[1333] 2. Terminal

[1334] The terminal is an input device for in-room and remote participants and has the following main functions:

[1335] Camera and microphone: Captures video and audio of meeting participants and sends them to the server.

[1336] Display device: Displays materials and information sent from the server on the conference screen.

[1337] Emotion recognition: Analyze participants' facial expressions and tone of voice to obtain emotional data.

[1338] 3. Users

[1339] A user is an individual who participates in a conference and performs the following actions:

[1340] Voice questions: Ask voice questions during the meeting.

[1341] Viewing materials: View materials and information displayed on the device.

[1342] 4. Emotion Engine

[1343] The emotion engine is a component that recognizes and analyzes the user's emotions.

[1344] Emotion analysis: Analyze user emotions in real time through facial expression recognition and voice analysis.

[1345] Data provision: Analysis results are sent to the server and used to dynamically change materials and information.

[1346] Program processing explanation

[1347] The system's programs are processed in cooperation with the server, terminal, user, and emotion engine.

[1348] Facial recognition and participant list creation

[1349] 1. The device captures video from within the conference room using a camera and transmits it to the server in real time.

[1350] 2. The server applies a facial recognition algorithm to identify the attendee's face and match it with a database.

[1351] 3. The server generates an attendee list based on the matching results and sends it to the terminal.

[1352] 4. The terminal displays the list of conference participants on the screen.

[1353] Real-time question processing and answer generation

[1354] 1. The user speaks a question, and the device captures the audio and sends it to the server.

[1355] 2. The server uses a speech recognition engine to convert the voice data into text and analyze the content.

[1356] 3. The server searches and generates relevant materials and information from the database.

[1357] 4. The server sends the generated materials to the terminal, which displays them on the conference screen.

[1358] Sentiment analysis and information provision

[1359] 1. The device captures the user's facial expressions and tone of voice using an emotion engine and analyzes them in real time.

[1360] 2. The emotion engine sends the acquired emotion data to the server.

[1361] 3. The server dynamically changes the information provided based on the emotion data and updates materials and information as necessary.

[1362] Post-conference analysis and solution presentation

[1363] 1. The server analyzes the data (voice, text, materials, emotional data) accumulated during the meeting.

[1364] 2. The server uses this data to categorize the issues discussed and conduct a detailed analysis.

[1365] 3. The server compiles solutions and prediction results for each problem into a report, sends it to the terminal, and shares it with the meeting participants.

[1366] Specific examples

[1367] For example, consider a case where a company holds a meeting to discuss marketing strategies for a new product.

[1368] Step 1: When the meeting starts, the device captures the video of the meeting room, and the server identifies all attendees and displays the participant list.

[1369] Step 2: When a user asks, "What is the target market for the new product?" the device captures the voice, the server converts it into text, and generates and displays relevant marketing materials.

[1370] Step 3: During the meeting, if the emotion engine senses the user's doubts or concerns, the server updates the materials and provides additional information.

[1371] Step 4: After the meeting ends, the server analyzes the discussion content and emotional data, generates a report presenting issues and solutions such as "reviewing marketing strategies" and "optimizing advertising budgets," and shares this report with meeting participants via their devices.

[1372] In this way, the present invention makes it possible to provide appropriate information in real time, dynamically change materials according to the emotions of participants, and also realizes the analysis of problems and the presentation of solutions after the meeting has ended.

[1373] The processing flow will be explained below.

[1374] Step 1:

[1375] The device captures video data from within the conference room using a camera and transmits it to the server in real time.

[1376] Step 2:

[1377] The server receives the captured video data and applies a facial recognition algorithm to identify the faces of the meeting participants. Specifically, it uses a face detection model (e.g., Haar Cascade or MTCNN) to identify facial features, then extracts and matches them with face data in a database.

[1378] Step 3:

[1379] The server generates a list of identified conference participants based on the matching result and transmits the generated list to the terminal.

[1380] Step 4:

[1381] The device displays the list of conference participants received on the screen. For example, "Yamada Taro - Project Manager" or "Sato Hanako - Engineer."

[1382] Step 5:

[1383] The user speaks a question, for example, "What is the target market for the new product?"

[1384] Step 6:

[1385] The terminal captures the voice data of the question with a microphone device and transmits the captured voice data to the server.

[1386] Step 7:

[1387] The server converts the voice data into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[1388] Step 8:

[1389] The server analyzes the textual question using natural language processing (NLP) technology, for example, to extract keywords such as "target market" from the question.

[1390] Step 9:

[1391] The server searches the database and generates materials and information related to the query (for example, presentation slides on marketing strategies).

[1392] Step 10:

[1393] The server transmits the generated materials to the terminal, which then displays them on the conference screen.

[1394] Step 11:

[1395] The device captures the user's facial expressions and tone of voice using an emotion engine and analyzes them in real time.

[1396] Step 12:

[1397] The emotion engine sends the acquired emotion data to the server. For example, if the user feels anxious or suspicious about a question, that emotion is sent to the server.

[1398] Step 13:

[1399] The server dynamically changes the information provided based on the emotion data, and searches and generates additional materials and information from the database as needed.

[1400] Step 14:

[1401] The server generates additional materials and sends them to the terminal, which then displays them on the conference screen. For example, if the user is feeling anxious, detailed statistical data or additional explanatory materials can be displayed.

[1402] Step 15:

[1403] After the meeting ends, the server analyzes the data accumulated during the meeting (voice, text, materials, and emotional data).

[1404] Step 16:

[1405] Based on the analyzed data, the server categorizes the issues discussed, such as "reviewing marketing strategies" or "optimizing advertising budgets."

[1406] Step 17:

[1407] The server performs a detailed analysis of each category and considers solutions based on relevant data.

[1408] Step 18:

[1409] The server compiles solutions and prediction results for each problem into a report.

[1410] Step 19:

[1411] The server sends the generated report to the terminal and shares it with the conference participants through the terminal. For example, after the conference, a report on the proposal for "optimizing the advertising budget" and its feasibility is shared.

[1412] This series of steps allows for efficient management of the progress of meetings, provision of appropriate information in real time, and post-meeting analysis of issues and presentation of solutions.It also enables dynamic changes to materials based on the emotions of participants.

[1413] Example 2

[1414] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1415] Conventional conference systems were able to convert participants' questions into text using facial recognition and voice recognition, and to search for and generate materials. However, they lacked the means to analyze participants' emotions in real time and dynamically update materials and information based on that. Another issue was the inability to provide predictive results through data analysis after the meeting. This could result in meetings not progressing smoothly and participants' satisfaction declining.

[1416] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for identifying the faces of the conference participants, means for generating a list of the identified conference participants, means for converting the content of the questions of the conference participants into text in real time using voice recognition technology, means for searching for and generating materials and information related to the content of the questions, means for capturing and analyzing emotional data of the conference participants, means for dynamically changing the information provided based on the emotional data and updating the materials and information as necessary, and means for categorizing issues discussed during the conference and presenting analysis results and solutions. This makes it possible to identify the faces and content of the questions of the conference participants as well as dynamically provide information according to the emotions of the participants, thereby realizing smooth progress of the conference and follow-up.

[1417] "Conference participant" refers to an individual who attends a conference and is a user of the system.

[1418] "Means for identifying faces" refers to a device or process that uses a facial recognition algorithm to identify conference participants based on camera footage.

[1419] A "facial recognition algorithm" is a set of mathematical formulas and programs that use image processing technology to recognize and identify individual faces.

[1420] "Means for generating a list of conference participants" refers to a device or process that organizes information about identified conference participants and displays or records it in list form.

[1421] "Voice recognition technology" is a technology that analyzes human voice and converts the content into text data.

[1422] "Means for converting questions into text" refers to a device or process that uses speech recognition technology to convert voice questions from conference participants into text data in real time.

[1423] "Means for retrieving and generating materials and information" refers to a device or process that retrieves the required materials and information from a database and generates them in an appropriate format.

[1424] "Means for capturing and analyzing emotional data" refers to a device or process that uses a camera or microphone to capture the facial expressions and tone of voice of meeting participants and analyzes them using an emotion analysis algorithm.

[1425] "Means for dynamically changing information provision based on emotional data" refers to a device or process that updates or changes the materials or information provided in real time based on analyzed emotional data.

[1426] "Means for categorizing issues and presenting analysis results and solutions" refers to a device or process that analyzes data from the content discussed during a meeting, categorizes it, and then proposes the analysis results and solutions based on them.

[1427] "Means for generating prediction results" refers to a device or process that analyzes the data accumulated after the meeting and generates results to predict future developments.

[1428] MODE FOR CARRYING OUT THE INVENTION

[1429] This system combines an automatic announcer that recognizes the faces of conference participants and processes their questions in real time with an emotion engine that recognizes and analyzes user emotions. This system supports the smooth progress of conferences, provides information according to the participants' emotions, and efficiently follows up after the conference.

[1430] Overall system configuration

[1431] The system mainly includes three main components and one additional component: a server, a terminal, a user, and an emotion engine.

[1432] 1. Server

[1433] The server is the center of the system and has the following main functions:

[1434] Facial Identification: Uses OpenCV to identify the faces of meeting participants from camera footage and match them with a pre-registered database.

[1435] Question content analysis: The user's question is converted into text using voice recognition technology (Google Cloud Speech-to-Text), and the content is analyzed.

[1436] Material search: Search and generate the necessary materials and information from the database.

[1437] Issue analysis: Analyze the data accumulated during the meeting and classify and analyze the issues discussed by category.

[1438] Solution Proposal: Based on the analysis results, we provide future solutions and predicted results.

[1439] Sentiment data analysis: Receive data from the sentiment engine and use it for follow-up during and after the meeting.

[1440] 2. Terminal

[1441] The terminal is an input device for in-room and remote participants and has the following main functions:

[1442] Camera and microphone: Captures video and audio of meeting participants and sends them to the server.

[1443] Display device: Displays materials and information sent from the server on the conference screen.

[1444] Emotion recognition: Analyze participants' facial expressions and tone of voice to obtain emotional data.

[1445] 3. Users

[1446] A user is an individual who participates in a conference and performs the following actions:

[1447] Voice questions: Ask voice questions during the meeting.

[1448] Viewing materials: View materials and information displayed on the device.

[1449] 4. Emotion Engine

[1450] The emotion engine is a component that recognizes and analyzes the user's emotions.

[1451] Emotion Analysis: Uses Microsoft Azure Emotion API to analyze user emotions in real time through facial expression recognition and voice analysis.

[1452] Data provision: Analysis results are sent to the server and used to dynamically change materials and information.

[1453] Specific examples

[1454] For example, consider a case where a company holds a meeting to discuss marketing strategies for a new product.

[1455] Step 1: At the start of the meeting, the device captures the video of the meeting room, and the server identifies all attendees. A participant list is created and displayed on the device.

[1456] Step 2: When a user asks, "What is the target market for the new product?", the device captures the voice, the server converts the voice into text, and generates and displays related materials.

[1457] Step 3: During the meeting, if the emotion engine senses the user's doubts or concerns, the server updates the materials and provides additional information.

[1458] Step 4: After the meeting ends, the server analyzes the discussion content and emotional data, generates a report presenting issues and solutions such as "reviewing marketing strategies" and "optimizing advertising budgets," and shares this report with meeting participants via their devices.

[1459] These functions are important for ensuring smooth progress in meetings and providing appropriate information according to the emotions of participants.

[1460] Prompt Sentence Examples

[1461] Describe how your system will respond if a user asks, "What is the target market for the new product?"

[1462] As described above, the present invention makes it possible to provide appropriate information in real time, dynamically change materials in response to the emotions of participants, and also to analyze issues and present solutions after the meeting has ended.

[1463] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1464] Step 1: Capture footage

[1465] Input: Camera video sent from the device.

[1466] Processing: The device captures video in real time through the camera in the conference room and sends the data to the server.

[1467] Output: Captured video data.

[1468] Step 2: Facial Recognition and Matching

[1469] Input: Captured video data.

[1470] Processing: The server runs a facial recognition algorithm using OpenCV to identify participants' faces from the video data and match them with pre-registered facial data in a database.

[1471] Output: A list of identified conference participants.

[1472] Step 3: Generate a participant list

[1473] Input: A list of identified conference participants.

[1474] Processing: The server generates this list and sends it to the terminal.

[1475] Output: Participant list sent to the device.

[1476] Step 4: View the Participant List

[1477] Input: The participant list sent to the device.

[1478] Processing: The terminal displays the received participant list on the conference screen. For example, names such as "Taro Tanaka" and "Hanako Sato" are displayed in the list.

[1479] Output: Participant list displayed on the conference screen.

[1480] Step 5: Capture audio

[1481] Input: User's spoken question.

[1482] Processing: During a meeting, a user asks a question by voice, such as "Where are the marketing materials?" The device captures the voice with its microphone and sends it to the server as audio data.

[1483] Output: The audio data sent to the server.

[1484] Step 6: Speech to Text

[1485] Input: The audio data sent to the server.

[1486] Processing: The server uses Google Cloud Speech-to-Text to convert the audio data into text. For example, the audio is converted into text, such as "Where are the marketing materials?"

[1487] Output: The question converted to text.

[1488] Step 7: Parsing the Question

[1489] Input: The question converted to text.

[1490] Processing: The server analyzes the textual question and searches the database for relevant information and materials.

[1491] Output: Search results for related materials.

[1492] Step 8: Find and generate relevant materials

[1493] Input: Search results for related materials.

[1494] Processing: The server generates the necessary materials based on the search results, for example preparing a PDF file of marketing materials.

[1495] Output: The generated material.

[1496] Step 9: Viewing the material

[1497] Input: Generated materials.

[1498] Processing: The server sends the generated materials to the terminal, which displays them on the conference screen. The user can view the displayed materials.

[1499] Output: Materials displayed on the meeting screen.

[1500] Step 10: Capturing Emotional Data

[1501] Input: Facial expressions and tone of voice of meeting participants.

[1502] Processing: The device uses a camera and microphone to capture emotional data such as the user's facial expressions and tone of voice in real time.

[1503] Output: Captured emotion data.

[1504] Step 11: Analyze the emotion data

[1505] Input: Captured emotion data.

[1506] Processing: An emotion engine (e.g., Microsoft Azure Emotion API) analyzes the captured data and determines the user's emotion. For example, it outputs an analysis result such as "The user looks anxious."

[1507] Output: Parsed emotion data.

[1508] Step 12: Dynamic information provision and documentation updates

[1509] Input: Parsed emotion data.

[1510] Processing: The emotion engine sends the analysis results to the server. The server then provides dynamic information based on the emotion data, adapting to the situation. For example, if the user appears anxious, the server provides additional materials with additional explanations. The device then displays the latest materials on the conference screen.

[1511] Output: Updated documentation and information.

[1512] Step 13: Data collection and analysis

[1513] Input: Audio, text, materials, emotional data, etc. accumulated during the meeting.

[1514] Processing: The server centralizes this data and performs detailed analysis.

[1515] Output: Analysis results.

[1516] Step 14: Categorize and analyze the issue

[1517] Input: Analysis results.

[1518] Processing: Based on the analysis results, the server categorizes the issues discussed and conducts a detailed analysis.

[1519] Output: A list of issues sorted by category.

[1520] Step 15: Generate and share your report

[1521] Input: A list of issues sorted by category.

[1522] Processing: The server generates a report that describes the solutions and prediction results for each problem. The report is sent to the terminal and can be viewed by the meeting participants.

[1523] Output: Report shared with meeting participants.

[1524] (Application example 2)

[1525] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1526] Modern factories are required to immediately solve problems faced by workers and improve productivity. However, conventional systems lack real-time problem-solving capabilities and support that adapts to the emotions of workers. As a result, work efficiency declines and work errors become more likely to occur. The present invention aims to provide a real-time problem-solving assistant system specialized for factory environments to solve these problems.

[1527] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for identifying the faces of factory workers, means for generating a list of identified factory workers, means for converting the contents of questions from factory workers into text in real time using voice recognition technology, means for searching and generating procedures and information related to the contents of the questions, and means for classifying problems that occur during factory work into categories and presenting analysis results and solutions. This makes it possible to quickly solve problems faced by factory workers and improve work efficiency.

[1528] "Conference participant" refers to any person participating in a conference.

[1529] "Means for identifying faces" refers to technology that uses a camera or facial recognition algorithm to recognize the face of a target person and identify that person.

[1530] The "means for generating a list of identified conference participants" is a technology for creating a list of participants based on information about conference participants obtained by face recognition.

[1531] "Speech recognition technology" is a technology that converts human speech into text data in real time.

[1532] "Means for searching and generating materials and information" refers to technology that searches for the necessary materials and information from related databases and information sources based on the content of a user's question and provides it.

[1533] "Means of categorizing issues and presenting analysis results and solutions" is a technique for organizing and classifying problems and issues discussed during meetings or work, and proposing solutions to them.

[1534] "Factory worker" refers to a worker who is in charge of work in a factory.

[1535] "Means for identifying the faces of factory workers" refers to technology that uses cameras and facial recognition algorithms to recognize the faces of target workers and identify them.

[1536] The "means for generating a list of identified factory workers" is a technology for creating a list of on-site workers based on information about factory workers obtained through facial recognition.

[1537] "Means for converting the content of questions posed by factory workers into text in real time using voice recognition technology" refers to technology for converting the voices spoken by factory workers into text in real time.

[1538] The "means for searching and generating procedures and information related to the content of a question" is a technology that searches a database for and provides relevant work procedures and information based on the content of a question posed by a factory worker.

[1539] "Means of categorizing problems that occur during factory work and presenting analysis results and solutions" is a technique for organizing and classifying troubles and problems that occur during work and proposing solutions to them.

[1540] This invention is a real-time problem-solving assistant system for improving work efficiency in a factory environment. This system mainly consists of a server, a terminal, a user, and an emotion engine.

[1541] Server Features

[1542] The server is the center of the system and has the following important functions:

[1543] Facial recognition: Identify the faces of factory workers from camera footage and compare them with a pre-registered database. Uses the dlib library and OpenCV as facial recognition algorithms.

[1544] Question content analysis: The questions from the workers are converted into text using speech recognition technology, and the content is analyzed. This is done using the speech_recognition library.

[1545] Document search: Search and generate the necessary procedures and information from a database that stores work procedures and equipment information within the factory.

[1546] Problem analysis: Analyze the data accumulated during the work and classify and analyze the problems that arise by category.

[1547] Solution Proposal: Based on the analysis results, we provide future solutions and predicted results.

[1548] Emotion data analysis: Receive data from the emotion engine and use it for follow-up during and after work.

[1549] Device Features

[1550] The terminal is an input device used by factory workers and has the following functions:

[1551] Camera and microphone: Captures video and audio of factory workers and sends them to a server.

[1552] Display device: The procedure and information sent from the server are displayed on the work screen. Display devices include smart glasses and smartphones.

[1553] Emotion Recognition: Analyzes the facial expressions and tone of voice of workers to obtain emotional data. Emotion recognition is performed using the Affectiva API.

[1554] User Actions

[1555] The user refers to a factory worker and performs the following actions:

[1556] Voice questions: Ask questions by voice as you work.

[1557] View Instructions: View instructions and information displayed on your device.

[1558] Emotion Engine Functions

[1559] The emotion engine is a component that recognizes and analyzes the emotions of workers.

[1560] Emotion analysis: Analyze workers' emotions in real time through facial expression recognition and voice analysis.

[1561] Data provision: The analysis results are sent to the server and used to dynamically change procedures and information.

[1562] Specific examples

[1563] For example, consider a situation where a problem occurs during the operation of a machine in a factory.

[1564] Step 1: When work begins, the device captures the camera image, and the server identifies the worker. A list of workers is displayed.

[1565] Step 2: When the worker asks, "Why is the machine stuck?" the device captures the voice, the server transcribes it, and generates the relevant repair instructions.

[1566] Step 3: If the emotion engine senses confusion or anxiety from the worker during the work, the server updates the displayed repair procedure with additional information to support the worker.

[1567] Step 4: After the work is completed, the server analyzes the accumulated data and generates a report that presents solutions such as "machine maintenance methods" and "future improvements," which is shared with the worker via their terminal.

[1568] Prompt Sentence Examples

[1569] "Devant a system that describes the situation when a maintenance issue occurs in the factory and generates real-time solutions to specific questions. Also include the ability to understand worker sentiment and provide follow-up accordingly."

[1570] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1571] Step 1:

[1572] Input: A factory worker activates smart glasses and camera footage is captured.

[1573] Processing (data processing and calculation): The device (smart glasses) captures the video of the factory worker and sends it to the server. The server uses OpenCV and dlib libraries to perform facial recognition and compares it with existing data in the database.

[1574] Output: A list of identified factory workers is generated and sent to the terminal.

[1575] How it works: The server applies a facial recognition algorithm to identify factory workers, creates a list, and displays it on the device.

[1576] Step 2:

[1577] Input: A factory worker asks a voiceover, "Why is the machine stuck?"

[1578] Processing (data processing and calculation): The device captures the voice and sends it to the server. The server converts the voice data into text using the speech_recognition library and analyzes the question.

[1579] Output: Repair procedures and information related to the query are searched for and generated from the database.

[1580] Specific operation: The server searches the database based on the text of the question, and sends relevant repair procedures and information to the terminal, where they are displayed.

[1581] Step 3:

[1582] Input: As the factory worker continues to follow the displayed steps, the emotion engine captures their facial expressions and tone of voice.

[1583] Processing (data processing and calculation): The device captures emotion data and sends it to the emotion engine in real time. The emotion engine analyzes emotions using the Affectiva API.

[1584] Output: The captured emotion data is sent to a server, which provides additional information or assistance if needed.

[1585] Specific operation: The server receives the sentiment analysis results and provides further support information if the worker is experiencing difficulties.

[1586] Step 4:

[1587] Input: The worker completes the repair procedure and the data on the work is sent from the terminal to the server.

[1588] Processing (data processing and calculation): The server analyzes the data (voice, text, emotion data) accumulated during the work and categorizes the problems that arise. Based on the analysis results, it generates solutions and predictions.

[1589] Output: A report containing the analysis results and solutions is generated and sent to the device.

[1590] Specific operation: The server generates a report and sends it to the terminal to be shared with workers and managers.

[1591] Step 5:

[1592] Input: Server-generated reports are reviewed by operators and administrators.

[1593] Processing (data processing and calculation): The report is displayed on the terminal, and suggestions for improvements and maintenance methods for the next work are made as needed.

[1594] Output: Next work plan and improvement suggestions are provided as reference for workers and managers.

[1595] Specific action: Workers and managers use the report to plan the next work.

[1596] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1597] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1598] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1599] [Fourth embodiment]

[1600] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1601] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1602] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1603] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1604] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1605] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1606] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1607] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1608] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1609] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1610] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1611] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1612] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1613] This invention provides an automatic conference announcer system that recognizes the faces of conference participants and processes questions in real time. This system manages the conference from start to finish, instantly generating and providing necessary materials and information, and also analyzes issues and presents solutions after the conference ends.

[1614] Overall system configuration

[1615] The system comprises three main components: a server, a terminal, and a user.

[1616] 1. Server

[1617] The server is the heart of the system and has the following main functions:

[1618] Facial recognition: The faces of meeting participants are identified from camera footage and compared with a pre-registered database.

[1619] Question content analysis: The user's question is converted into text using voice recognition technology, and the content is analyzed.

[1620] Material search: Search and generate the necessary materials and information from the database.

[1621] Issue analysis: Analyze the data accumulated during the meeting and classify and analyze the issues discussed by category.

[1622] Solution Proposal: Based on the analysis results, we provide future solutions and predicted results.

[1623] 2. Terminal

[1624] The terminal is an input device for in-room and remote participants and has the following main functions:

[1625] Camera and microphone: Captures video and audio of meeting participants and sends them to the server.

[1626] Display device: Displays materials and information sent from the server on the conference screen.

[1627] 3. Users

[1628] A user is an individual who participates in a conference and performs the following actions:

[1629] Voice questions: Ask voice questions during the meeting.

[1630] Viewing materials: View materials and information displayed on the device.

[1631] Program processing explanation

[1632] The processing of the programs in this system is mainly carried out in cooperation between the server, the terminal, and the user.

[1633] Facial recognition and participant list creation

[1634] 1. The device captures video of the conference room with its camera and sends it to the server.

[1635] 2. The server uses a facial recognition algorithm to identify attendees' faces and compare them with a registered database. Based on the results of the comparison, it generates an attendee list, which is sent to the device and displayed on the screen.

[1636] Real-time question processing and answer generation

[1637] 1. The user speaks a question, and the device captures the audio and sends it to the server.

[1638] 2. The server uses voice recognition technology to convert the question into text and analyzes the content.

[1639] 3. The server searches the database for materials and information related to the question, generates the necessary materials, and sends them to the terminal to display on the screen.

[1640] Post-conference analysis and solution presentation

[1641] 1. The server analyzes the data (audio, text, materials) accumulated during the meeting and extracts the content discussed.

[1642] 2. The server classifies the extracted issues into categories and performs detailed analysis.

[1643] 3. The server generates solutions and prediction results for each problem, compiles them into a report, and sends it to the terminal to share with the meeting participants.

[1644] Specific examples

[1645] For example, consider a case where a company holds a meeting to discuss marketing strategies for a new product.

[1646] Step 1: When the meeting starts, the device captures the video of the meeting room, and the server identifies all attendees and displays a list of participants.

[1647] Step 2: When a user asks, "What is the target market for the new product?" the device captures the voice, the server converts it into text, and generates and displays relevant marketing materials.

[1648] Step 3: After the meeting, the server analyzes the content discussed and generates a report presenting issues and solutions, such as "reviewing marketing strategies" and "optimizing advertising budgets," which is shared with meeting participants via their devices.

[1649] In this way, the present invention allows for the provision of appropriate information in real time and efficient meeting follow-up.

[1650] The processing flow will be explained below.

[1651] Step 1:

[1652] The device captures video data from within the conference room using a camera and transmits it to the server in real time.

[1653] Step 2:

[1654] The server receives the captured video data and applies a facial recognition algorithm to identify the faces of the meeting participants. Specifically, it uses a face detection model (e.g., Haar Cascade or MTCNN) to identify facial features, then extracts and matches them with face data in a database.

[1655] Step 3:

[1656] The server generates a list of identified conference participants based on the matching result and transmits the generated list to the terminal.

[1657] Step 4:

[1658] The device displays the list of conference participants received on the screen. For example, "Yamada Taro - Project Manager" or "Sato Hanako - Engineer."

[1659] Step 5:

[1660] The user speaks a question, for example, "What is the target market for the new product?"

[1661] Step 6:

[1662] The terminal captures the voice data of the question with a microphone device and transmits the captured voice data to the server.

[1663] Step 7:

[1664] The server converts the voice data into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[1665] Step 8:

[1666] The server analyzes the textual question using natural language processing (NLP) technology, for example, to extract keywords such as "target market" from the question.

[1667] Step 9:

[1668] The server searches the database and generates materials and information related to the query (for example, presentation slides on marketing strategies).

[1669] Step 10:

[1670] The server transmits the generated materials to the terminal, which then displays them on the conference screen.

[1671] Step 11:

[1672] After the meeting ends, the server analyzes the data (audio, text, and materials) accumulated during the meeting.

[1673] Step 12:

[1674] Based on the analyzed data, the server categorizes the issues discussed, such as "reviewing marketing strategies" or "optimizing advertising budgets."

[1675] Step 13:

[1676] The server performs a detailed analysis of each category and considers solutions based on relevant data.

[1677] Step 14:

[1678] The server compiles solutions and prediction results for each problem into a report.

[1679] Step 15:

[1680] The server sends the generated report to the terminal and shares it with the conference participants through the terminal.

[1681] This series of steps enables efficient management of the meeting progress, provision of appropriate information in real time, and post-meeting analysis of issues and presentation of solutions.

[1682] Example 1

[1683] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1684] During a meeting, it is necessary to identify participants' faces, process questions in real time, search for and generate relevant materials, analyze the issues discussed, and present solutions. However, it is difficult to perform these tasks efficiently and automatically. It is also equally difficult to quickly analyze the content discussed after the meeting and present solutions afterward. Conventional systems only automate part of these processes, resulting in a decrease in overall efficiency.

[1685] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1686] In this invention, the server includes means for identifying the faces of conference participants, means for generating a list of identified conference participants, means for converting the contents of questions asked by the conference participants into text in real time using voice recognition technology, means for searching for and generating materials and information related to the contents of the questions, means for categorizing issues discussed during the conference and presenting analysis results and solutions, means for transmitting the generated materials and information to terminals and displaying them on the conference screen, and means for analyzing accumulated data after the conference ends, extracting and categorizing the contents discussed, and generating solutions. This makes it possible to automate and efficiently manage the overall process of conference progress and follow-up.

[1687] A "conference participant" is an individual participating in a conference.

[1688] A "means for identifying faces" is a technology or device for analyzing camera footage and recognizing the faces of individual conference participants.

[1689] The "means for generating a list of conference participants" is a technology or device that creates a list of people participating in a conference based on the results of face identification.

[1690] "Speech recognition technology" is a technology that converts voice data into text data in real time.

[1691] "Means for converting questions into text in real time using voice recognition technology" refers to a technology or device that captures questions uttered by conference participants as voice data and instantly converts it into text.

[1692] "Means for searching and generating materials and information" refers to technology or equipment that searches for necessary materials and information from a database and generates them on request.

[1693] A "means for categorizing issues" is a technique or device that separates the content discussed during a meeting into multiple categories.

[1694] "Means for presenting analysis results and solutions" refers to a technique or device that analyzes the content of the discussion and derives and presents solutions based on the results.

[1695] The "means for transmitting generated materials and information to the terminal" refers to a technology or device for transmitting various materials and information generated by the server to the terminal in the conference room.

[1696] The "means for displaying on the conference screen" refers to a technique or device for displaying materials and information sent to the terminal on the conference screen.

[1697] The "means for analyzing data accumulated after the end of a meeting" refers to a technology or device for analyzing data such as audio, text, and materials collected after the end of a meeting.

[1698] "Means for extracting and classifying the content discussed and generating solutions" refers to a technology or device that analyzes the content discussed during a meeting, extracts important elements, classifies them by category, and generates solutions.

[1699] This invention provides an automatic conference announcer system that recognizes the faces of conference participants and processes questions in real time. This system manages the conference from start to finish, instantly generating and providing necessary materials and information, and also analyzes issues and presents solutions after the conference ends.

[1700] Overall system configuration

[1701] The system comprises three main components: a server, a terminal, and a user.

[1702] 1. Server

[1703] The server is the heart of the system and has the following main functions:

[1704] Facial recognition: The server receives the video footage from the conference room camera sent from the device, identifies attendees using a facial recognition algorithm, and compares them with a registered database. Technologies used include OpenCV and Dlib.

[1705] Question analysis: The server receives the data captured by the device and sends the voice question from the user, and converts the question into text using speech recognition technology. This can be done using the Google Speech-to-Text API. The text question is then analyzed using natural language processing technology (e.g., SpaCy).

[1706] Material search: The server uses a search engine such as Elasticsearch to search and generate the necessary materials and information from the database.

[1707] Issue analysis: The server analyzes the data accumulated during the meeting and categorizes and analyzes the issues discussed using natural language processing toolkits (e.g., NLTK, BERT).

[1708] Solution Proposal: The server generates future solutions and predictions based on the analysis results, using a generative AI model (e.g., GPT-4) to provide the optimal solution.

[1709] 2. Terminal

[1710] The terminal is an input device for in-room and remote participants and has the following main functions:

[1711] Camera and microphone: The device captures the video and audio of meeting participants in real time and sends them to the server.

[1712] Display device: The terminal receives materials and information sent from the server and displays them on the conference screen. This can be an electronic whiteboard or video conferencing software (e.g., Zoom, Microsoft Teams).

[1713] 3. Users

[1714] A user is an individual who participates in a conference and performs the following actions:

[1715] Voice questions: Users ask questions by voice during the conference, which are captured by the terminal and sent to the server.

[1716] Viewing materials: Users view materials and information displayed on their terminals and obtain appropriate information as the meeting progresses.

[1717] Program processing explanation

[1718] Facial recognition and participant list creation

[1719] The device captures video of the conference room with its camera and sends it to the server, where the server uses a facial recognition algorithm to identify attendees' faces and compare them with a registered database. Based on the results of the comparison, a list of attendees is generated and sent to the device, where it is displayed on the screen.

[1720] Real-time question processing and answer generation

[1721] The user speaks a question, and the device captures the voice and sends it to the server. The server uses voice recognition technology to convert the question into text and analyzes it. The server then searches a database for materials and information related to the question and generates the necessary materials. These are then sent to the device and displayed on the screen.

[1722] Post-conference analysis and solution presentation

[1723] The server analyzes the data (audio, text, and materials) accumulated during the meeting and extracts the content of the discussion. The server then categorizes the extracted issues and performs a detailed analysis. The server then generates solutions and predictions for each issue, compiles them into a report, and sends it to the device to share with the meeting participants.

[1724] Specific examples

[1725] For example, consider a case where a company holds a meeting to discuss marketing strategies for a new product.

[1726] Step 1: When the meeting starts, the device captures the video of the meeting room, and the server identifies all attendees and displays a list of participants.

[1727] Step 2: When a user asks, "What is the target market for the new product?" the device captures the voice, the server converts it into text, and generates and displays relevant marketing materials.

[1728] Step 3: After the meeting, the server analyzes the content discussed and generates a report presenting issues and solutions, such as "reviewing marketing strategies" and "optimizing advertising budgets," which is shared with meeting participants via their devices.

[1729] Examples of prompt statements

[1730] "Please display information about the target market for any new products discussed during the meeting. Also provide examples of past successful marketing strategies."

[1731] As described above, the present invention realizes appropriate information provision in real time and efficient conference follow-up.

[1732] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1733] Step 1:

[1734] The device captures video of the conference room with a camera and sends the data to the server in real time. This input data is a video stream, and each frame contains the faces of the conference participants. The server receives this data and performs preprocessing for face identification. Specifically, it analyzes the video data and detects and extracts faces from each frame.

[1735] Step 2:

[1736] The server applies a facial recognition algorithm (e.g., OpenCV or Dlib) to the preprocessed image data, extracts the features of the recognized face, and compares them with a pre-registered database. The matching results are output as the ID and name of the identified conference participant.

[1737] Step 3:

[1738] The server generates an attendee list based on the matching results. This list includes the name and related information of each conference participant. The generated attendee list is sent from the server to the terminal and displayed on the terminal's conference screen. This data output is displayed on the conference screen as text information.

[1739] Step 4:

[1740] The user speaks a question, and the device captures and sends the audio to the server. The input data is an audio file or a real-time audio stream. The server receives the data and performs preprocessing for speech recognition, such as noise reduction and speech normalization.

[1741] Step 5:

[1742] The server uses speech recognition technology (e.g., Google Speech-to-Text API) to convert the preprocessed voice data into text, and the output is the question in text format.

[1743] Step 6:

[1744] The server analyzes the text of the question using natural language processing (NLP) technology (e.g., SpaCy). As a result of this analysis, the intent of the question and related keywords are extracted.

[1745] Step 7:

[1746] The server searches the database for relevant materials and information based on the analysis results. This process uses a search engine such as Elasticsearch. The search results include links to related materials and summaries.

[1747] Step 8:

[1748] The server generates the necessary materials based on the search results and sends them to the terminal. The generated materials are displayed on the terminal's conference screen. This output data includes PDFs and presentation slides.

[1749] Step 9:

[1750] After the meeting ends, the server analyzes the accumulated meeting data (audio, text, and materials). It uses a natural language processing toolkit (e.g., NLTK, BERT) to extract and classify the issues discussed. The output is a list of the extracted issues and their categorization.

[1751] Step 10:

[1752] The server generates solutions to each problem based on the analysis results. Using a generative AI model (e.g., GPT-4), it outputs optimal solutions and predictions. These are sent to the terminal in the form of a report and shared with the meeting participants.

[1753] Step 11:

[1754] The device displays the received report to the meeting participants, allowing them to view the shared information, which allows for efficient follow-up after the meeting has ended.

[1755] In this way, the overall process of meeting progress and follow-up is detailed for each processing step.

[1756] (Application example 1)

[1757] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1758] Conventional conference systems have difficulty recognizing the faces of conference participants, processing questions in real time, generating materials instantly, and analyzing issues and presenting solutions after the conference has ended. Furthermore, in customer service at brick-and-mortar stores, there are challenges in effectively identifying customers' faces and providing personalized service, making it difficult to improve customer satisfaction. Furthermore, there is a need to analyze customer voices and instantly present in-store product information and inventory status. To solve these problems, a system capable of more efficient, real-time information processing and personalized response is needed.

[1759] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1760] In this invention, the server includes means for identifying the faces of conference participants, means for generating a list of identified conference participants, means for converting the content of questions from conference participants into text in real time using voice recognition technology, means for searching for and generating materials and information related to the content of the questions, means for categorizing issues discussed during the conference and presenting analysis results and solutions, means installed in smart glasses for identifying customers through entity recognition and providing personalized service, and means for analyzing the customer's voice and presenting product information and inventory status in the store. This enables appropriate information provision and efficient follow-up of issues in conferences, and enables real-time, personalized support to improve customer satisfaction in physical stores.

[1761] The "means for identifying the faces of conference participants" refers to a method of capturing the faces of conference participants using a camera and identifying individual participants using a facial recognition algorithm.

[1762] The "means for generating a list of identified conference participants" is a method for organizing and displaying information about conference participants identified by facial recognition in list form.

[1763] "Means for converting questions from conference participants into text in real time using speech recognition technology" refers to a method for converting speech spoken by conference participants into text data in real time using speech recognition technology.

[1764] "Means for searching and generating materials and information related to the question content" refers to a method for searching a database for related materials and information based on the analyzed question content, and generating and displaying them.

[1765] "Method of categorizing the issues discussed during the meeting and presenting the analysis results and solutions" is a method of organizing and analyzing the content discussed during the meeting by category, and presenting the analysis results and solutions based on that.

[1766] "Means installed on smart glasses for identifying customers through entity recognition and providing personalized service" refers to a method for identifying customers visiting a store using a camera and a recognition system installed in smart glasses and providing personalized service to the customer.

[1767] "Means for analyzing customer voice and presenting in-store product information and stock status" refers to a method for analyzing customer voice in real time and displaying and providing in-store product information and stock status based on the analysis results.

[1768] The present invention is a system including two main application examples: a conference system and a brick-and-mortar store system. This system is configured with a server, a terminal, and a user as its main components.

[1769] Conference System

[1770] server

[1771] The server has the following main functions:

[1772] 1. Facial Identification: Identify the faces of meeting participants from camera footage and compare them with a pre-registered database. OpenCV and TensorFlow are used for facial identification.

[1773] 2. Speech recognition: The user's question is converted into text using PyAudio and SpeechRecognition technology, and the content is analyzed.

[1774] 3. Document search: Search and generate documents and information related to the question from the database. This is done using SQLAlchemy.

[1775] 4. Issue analysis: Analyze the data accumulated during the meeting and classify and analyze the issues discussed by category.

[1776] 5. Solution Proposal: Based on the analysis results, provide future solutions and predicted results.

[1777] Terminal

[1778] The terminal has the following main features:

[1779] 1. Camera and microphone: Captures video and audio of meeting participants and sends them to the server.

[1780] 2. Display device: Displays materials and information sent from the server on the conference screen.

[1781] User

[1782] A user is an individual who participates in a conference and performs the following actions:

[1783] 1. Audio Questions: Ask questions by voice during the meeting.

[1784] 2. Viewing materials: View materials and information displayed on the terminal.

[1785] Physical store compatible system

[1786] Smart Glasses

[1787] The smart glasses have the following main features:

[1788] 1. Facial Recognition: Recognize the faces of customers who visit your physical store and identify repeat and new customers. OpenCV and TensorFlow are used for facial recognition.

[1789] 2. Speech Recognition: Analyze customer questions in real time and provide appropriate information, using PyAudio and SpeechRecognition technology.

[1790] 3. Product information display: Display product information and stock status based on customer questions. Use SQLAlchemy for database management.

[1791] server

[1792] The server has the following main functions:

[1793] 1. Facial recognition: Identify customers using video transmitted from smart glasses.

[1794] 2. Speech recognition: Converts the customer's voice into text and analyzes its content.

[1795] 3. Information search: Search and generate in-store product information and inventory status based on customer questions.

[1796] 4. History management: Maintain customer visit and purchase history to provide personalized service.

[1797] Specific examples

[1798] For example, if a repeat customer visits a store, the camera in the smart glasses will recognize the customer and respond with a voice message saying, "Welcome. Thank you for visiting again." If the customer asks, "Is this item in stock?", the voice recognition system will respond, "The item in question is currently in stock."

[1799] Also, if a participant asks during a meeting, "Is there a recent report on this issue?", the server will convert the speech into text, search for relevant materials, and display them.

[1800] Prompt Sentence Examples

[1801] "Generate program code for a smart glasses assistant that recognizes customer faces, greets returning and new customers appropriately, and provides inventory information and product recommendations based on customer questions. Use OpenCV and TensorFlow for face recognition, PyAudio and SpeechRecognition for voice recognition, and SQLAlchemy for database management."

[1802] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1803] Step 1:

[1804] Facial Recognition Processing

[1805] Server: Receives video footage of the conference room or store from the device and extracts facial features using OpenCV. Next, it uses TensorFlow's facial recognition algorithm to calculate facial features and compare them with facial data in the database.

[1806] Input: Video data from the device.

[1807] Output: A list of recognized meeting participants or customers.

[1808] Specific operation: The camera detects faces in each captured video frame, calculates features, and matches them with face information in a matching database.

[1809] Step 2:

[1810] Generate and display a list of conference participants

[1811] Server: Generates a list of identified conference participants based on facial recognition and sends the list to the terminal.

[1812] Input: Face identification results.

[1813] Output: A list of conference participants.

[1814] Specific operation: The recognized participant's name or ID is added to a list, sent to the device, and displayed.

[1815] Step 3:

[1816] Capture and transcribing voice questions

[1817] Terminal: The microphone captures the user's voice and sends it to the server.

[1818] Server: Converts received audio data into text using PyAudio and SpeechRecognition technology.

[1819] Input: Audio data from the user.

[1820] Output: Textualized question.

[1821] Specific operation: The user's voice is detected by the microphone, sent to the server in real time, and converted into text using voice recognition technology.

[1822] Step 4:

[1823] Searching for and generating materials and information

[1824] Server: Analyzes the textual content of the question, searches for and generates relevant materials and information from the database using SQLAlchemy, and sends the materials to the terminal.

[1825] Input: The text of the question.

[1826] Output: Related materials and information.

[1827] Specific operation: The database is searched using the question as a keyword, and relevant documents and information are extracted and sent to the terminal.

[1828] Step 5:

[1829] Categorizing problems and presenting solutions

[1830] Server: Analyzes questions and discussions accumulated during the meeting and categorizes them. Generates solutions for each category and sends detailed analysis results to the device.

[1831] Input: Questions and discussions during the meeting.

[1832] Output: Categorised challenges and their solutions.

[1833] Specific operation: The accumulated data is analyzed using natural language processing technology and classified into categories, and then a solution is generated using an AI model and sent to the terminal in report format.

[1834] Step 6:

[1835] Identifying and responding to customers in-store

[1836] Smart glasses: Capture the face of the customer and send it to the server. Based on the results of the server's matching, the glasses present the customer with appropriate greetings and information.

[1837] Input: Customer video data.

[1838] Output: Customer identification and response message.

[1839] How it works: The smart glasses' camera captures the customer's face and sends it to the server. When the recognition result is returned, a voice or visual response is given based on that data.

[1840] Step 7:

[1841] Analysis of customer voice questions and information presentation

[1842] Smart glasses: Capture customer voice questions with a microphone and send them to a server.

[1843] Server: Analyzes the voice data, searches for and generates product information and stock status based on the question, and sends that information to the smart glasses.

[1844] Input: Voice data from the customer.

[1845] Output: Product information and stock status.

[1846] Specific operation: The customer's voice is captured by a microphone and sent to a server, where it is analyzed using voice recognition technology, and the appropriate information is searched for in a database and sent to and displayed on the smart glasses.

[1847] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1848] This system combines an automatic announcer that recognizes the faces of conference participants and processes questions in real time with an emotion engine that recognizes the user's emotions. This system supports the smooth progress of conferences, provides information according to the participants' emotions, and efficiently follows up after the conference.

[1849] Overall system configuration

[1850] The system mainly includes three main components and one additional component: a server, a terminal, a user, and an emotion engine.

[1851] 1. Server

[1852] The server is the center of the system and has the following main functions:

[1853] Facial recognition: The faces of meeting participants are identified from camera footage and compared with a pre-registered database.

[1854] Question content analysis: The user's question is converted into text using voice recognition technology, and the content is analyzed.

[1855] Material search: Search and generate the necessary materials and information from the database.

[1856] Issue analysis: Analyze the data accumulated during the meeting and classify and analyze the issues discussed by category.

[1857] Solution Proposal: Based on the analysis results, we provide future solutions and predicted results.

[1858] Sentiment data analysis: Receive data from the sentiment engine and use it for follow-up during and after the meeting.

[1859] 2. Terminal

[1860] The terminal is an input device for in-room and remote participants and has the following main functions:

[1861] Camera and microphone: Captures video and audio of meeting participants and sends them to the server.

[1862] Display device: Displays materials and information sent from the server on the conference screen.

[1863] Emotion recognition: Analyze participants' facial expressions and tone of voice to obtain emotional data.

[1864] 3. Users

[1865] A user is an individual who participates in a conference and performs the following actions:

[1866] Voice questions: Ask voice questions during the meeting.

[1867] Viewing materials: View materials and information displayed on the device.

[1868] 4. Emotion Engine

[1869] The emotion engine is a component that recognizes and analyzes the user's emotions.

[1870] Emotion analysis: Analyze user emotions in real time through facial expression recognition and voice analysis.

[1871] Data provision: Analysis results are sent to the server and used to dynamically change materials and information.

[1872] Program processing explanation

[1873] The system's programs are processed in cooperation with the server, terminal, user, and emotion engine.

[1874] Facial recognition and participant list creation

[1875] 1. The device captures video from within the conference room using a camera and transmits it to the server in real time.

[1876] 2. The server applies a facial recognition algorithm to identify the attendee's face and match it with a database.

[1877] 3. The server generates an attendee list based on the matching results and sends it to the terminal.

[1878] 4. The terminal displays the list of conference participants on the screen.

[1879] Real-time question processing and answer generation

[1880] 1. The user speaks a question, and the device captures the audio and sends it to the server.

[1881] 2. The server uses a speech recognition engine to convert the voice data into text and analyze the content.

[1882] 3. The server searches and generates relevant materials and information from the database.

[1883] 4. The server sends the generated materials to the terminal, which displays them on the conference screen.

[1884] Sentiment analysis and information provision

[1885] 1. The device captures the user's facial expressions and tone of voice using an emotion engine and analyzes them in real time.

[1886] 2. The emotion engine sends the acquired emotion data to the server.

[1887] 3. The server dynamically changes the information provided based on the emotion data and updates materials and information as necessary.

[1888] Post-conference analysis and solution presentation

[1889] 1. The server analyzes the data (voice, text, materials, emotional data) accumulated during the meeting.

[1890] 2. The server uses this data to categorize the issues discussed and conduct a detailed analysis.

[1891] 3. The server compiles solutions and prediction results for each problem into a report, sends it to the terminal, and shares it with the meeting participants.

[1892] Specific examples

[1893] For example, consider a case where a company holds a meeting to discuss marketing strategies for a new product.

[1894] Step 1: When the meeting starts, the device captures the video of the meeting room, and the server identifies all attendees and displays the participant list.

[1895] Step 2: When a user asks, "What is the target market for the new product?" the device captures the voice, the server converts it into text, and generates and displays relevant marketing materials.

[1896] Step 3: During the meeting, if the emotion engine senses the user's doubts or concerns, the server updates the materials and provides additional information.

[1897] Step 4: After the meeting ends, the server analyzes the discussion content and emotional data, generates a report presenting issues and solutions such as "reviewing marketing strategies" and "optimizing advertising budgets," and shares this report with meeting participants via their devices.

[1898] In this way, the present invention makes it possible to provide appropriate information in real time, dynamically change materials according to the emotions of participants, and also realizes the analysis of problems and the presentation of solutions after the meeting has ended.

[1899] The processing flow will be explained below.

[1900] Step 1:

[1901] The device captures video data from within the conference room using a camera and transmits it to the server in real time.

[1902] Step 2:

[1903] The server receives the captured video data and applies a facial recognition algorithm to identify the faces of the meeting participants. Specifically, it uses a face detection model (e.g., Haar Cascade or MTCNN) to identify facial features, then extracts and matches them with face data in a database.

[1904] Step 3:

[1905] The server generates a list of identified conference participants based on the matching result and transmits the generated list to the terminal.

[1906] Step 4:

[1907] The device displays the list of conference participants received on the screen. For example, "Yamada Taro - Project Manager" or "Sato Hanako - Engineer."

[1908] Step 5:

[1909] The user speaks a question, for example, "What is the target market for the new product?"

[1910] Step 6:

[1911] The terminal captures the voice data of the question with a microphone device and transmits the captured voice data to the server.

[1912] Step 7:

[1913] The server converts the voice data into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[1914] Step 8:

[1915] The server analyzes the textual question using natural language processing (NLP) technology, for example, to extract keywords such as "target market" from the question.

[1916] Step 9:

[1917] The server searches the database and generates materials and information related to the query (for example, presentation slides on marketing strategies).

[1918] Step 10:

[1919] The server transmits the generated materials to the terminal, which then displays them on the conference screen.

[1920] Step 11:

[1921] The device captures the user's facial expressions and tone of voice using an emotion engine and analyzes them in real time.

[1922] Step 12:

[1923] The emotion engine sends the acquired emotion data to the server. For example, if the user feels anxious or suspicious about a question, that emotion is sent to the server.

[1924] Step 13:

[1925] The server dynamically changes the information provided based on the emotion data, and searches and generates additional materials and information from the database as needed.

[1926] Step 14:

[1927] The server generates additional materials and sends them to the terminal, which then displays them on the conference screen. For example, if the user is feeling anxious, detailed statistical data or additional explanatory materials can be displayed.

[1928] Step 15:

[1929] After the meeting ends, the server analyzes the data accumulated during the meeting (voice, text, materials, and emotional data).

[1930] Step 16:

[1931] Based on the analyzed data, the server categorizes the issues discussed, such as "reviewing marketing strategies" or "optimizing advertising budgets."

[1932] Step 17:

[1933] The server performs a detailed analysis of each category and considers solutions based on relevant data.

[1934] Step 18:

[1935] The server compiles solutions and prediction results for each problem into a report.

[1936] Step 19:

[1937] The server sends the generated report to the terminal and shares it with the conference participants through the terminal. For example, after the conference, a report on the proposal for "optimizing the advertising budget" and its feasibility is shared.

[1938] This series of steps allows for efficient management of the progress of meetings, provision of appropriate information in real time, and post-meeting analysis of issues and presentation of solutions.It also enables dynamic changes to materials based on the emotions of participants.

[1939] Example 2

[1940] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1941] Conventional conference systems were able to convert participants' questions into text using facial recognition and voice recognition, and to search for and generate materials. However, they lacked the means to analyze participants' emotions in real time and dynamically update materials and information based on that. Another issue was the inability to provide predictive results through data analysis after the meeting. This could result in meetings not progressing smoothly and participants' satisfaction declining.

[1942] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for identifying the faces of the conference participants, means for generating a list of the identified conference participants, means for converting the content of the questions of the conference participants into text in real time using voice recognition technology, means for searching for and generating materials and information related to the content of the questions, means for capturing and analyzing emotional data of the conference participants, means for dynamically changing the information provided based on the emotional data and updating the materials and information as necessary, and means for categorizing issues discussed during the conference and presenting analysis results and solutions. This makes it possible to identify the faces and content of the questions of the conference participants as well as dynamically provide information according to the emotions of the participants, thereby realizing smooth progress of the conference and follow-up.

[1943] "Conference participant" refers to an individual who attends a conference and is a user of the system.

[1944] "Means for identifying faces" refers to a device or process that uses a facial recognition algorithm to identify conference participants based on camera footage.

[1945] A "facial recognition algorithm" is a set of mathematical formulas and programs that use image processing technology to recognize and identify individual faces.

[1946] "Means for generating a list of conference participants" refers to a device or process that organizes information about identified conference participants and displays or records it in list form.

[1947] "Voice recognition technology" is a technology that analyzes human voice and converts the content into text data.

[1948] "Means for converting questions into text" refers to a device or process that uses speech recognition technology to convert voice questions from conference participants into text data in real time.

[1949] "Means for retrieving and generating materials and information" refers to a device or process that retrieves the required materials and information from a database and generates them in an appropriate format.

[1950] "Means for capturing and analyzing emotional data" refers to a device or process that uses a camera or microphone to capture the facial expressions and tone of voice of meeting participants and analyzes them using an emotion analysis algorithm.

[1951] "Means for dynamically changing information provision based on emotional data" refers to a device or process that updates or changes the materials or information provided in real time based on analyzed emotional data.

[1952] "Means for categorizing issues and presenting analysis results and solutions" refers to a device or process that analyzes data from the content discussed during a meeting, categorizes it, and then proposes the analysis results and solutions based on them.

[1953] "Means for generating prediction results" refers to a device or process that analyzes the data accumulated after the meeting and generates results to predict future developments.

[1954] MODE FOR CARRYING OUT THE INVENTION

[1955] This system combines an automatic announcer that recognizes the faces of conference participants and processes their questions in real time with an emotion engine that recognizes and analyzes user emotions. This system supports the smooth progress of conferences, provides information according to the participants' emotions, and efficiently follows up after the conference.

[1956] Overall system configuration

[1957] The system mainly includes three main components and one additional component: a server, a terminal, a user, and an emotion engine.

[1958] 1. Server

[1959] The server is the center of the system and has the following main functions:

[1960] Facial Identification: Uses OpenCV to identify the faces of meeting participants from camera footage and match them with a pre-registered database.

[1961] Question content analysis: The user's question is converted into text using voice recognition technology (Google Cloud Speech-to-Text), and the content is analyzed.

[1962] Material search: Search and generate the necessary materials and information from the database.

[1963] Issue analysis: Analyze the data accumulated during the meeting and classify and analyze the issues discussed by category.

[1964] Solution Proposal: Based on the analysis results, we provide future solutions and predicted results.

[1965] Sentiment data analysis: Receive data from the sentiment engine and use it for follow-up during and after the meeting.

[1966] 2. Terminal

[1967] The terminal is an input device for in-room and remote participants and has the following main functions:

[1968] Camera and microphone: Captures video and audio of meeting participants and sends them to the server.

[1969] Display device: Displays materials and information sent from the server on the conference screen.

[1970] Emotion recognition: Analyze participants' facial expressions and tone of voice to obtain emotional data.

[1971] 3. Users

[1972] A user is an individual who participates in a conference and performs the following actions:

[1973] Voice questions: Ask voice questions during the meeting.

[1974] Viewing materials: View materials and information displayed on the device.

[1975] 4. Emotion Engine

[1976] The emotion engine is a component that recognizes and analyzes the user's emotions.

[1977] Emotion Analysis: Uses Microsoft Azure Emotion API to analyze user emotions in real time through facial expression recognition and voice analysis.

[1978] Data provision: Analysis results are sent to the server and used to dynamically change materials and information.

[1979] Specific examples

[1980] For example, consider a case where a company holds a meeting to discuss marketing strategies for a new product.

[1981] Step 1: At the start of the meeting, the device captures the video of the meeting room, and the server identifies all attendees. A participant list is created and displayed on the device.

[1982] Step 2: When a user asks, "What is the target market for the new product?", the device captures the voice, the server converts the voice into text, and generates and displays related materials.

[1983] Step 3: During the meeting, if the emotion engine senses the user's doubts or concerns, the server updates the materials and provides additional information.

[1984] Step 4: After the meeting ends, the server analyzes the discussion content and emotional data, generates a report presenting issues and solutions such as "reviewing marketing strategies" and "optimizing advertising budgets," and shares this report with meeting participants via their devices.

[1985] These functions are important for ensuring smooth progress in meetings and providing appropriate information according to the emotions of participants.

[1986] Prompt Sentence Examples

[1987] Describe how your system will respond if a user asks, "What is the target market for the new product?"

[1988] As described above, the present invention makes it possible to provide appropriate information in real time, dynamically change materials in response to the emotions of participants, and also to analyze issues and present solutions after the meeting has ended.

[1989] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1990] Step 1: Capture footage

[1991] Input: Camera video sent from the device.

[1992] Processing: The device captures video in real time through the camera in the conference room and sends the data to the server.

[1993] Output: Captured video data.

[1994] Step 2: Facial Recognition and Matching

[1995] Input: Captured video data.

[1996] Processing: The server runs a facial recognition algorithm using OpenCV to identify participants' faces from the video data and match them with pre-registered facial data in a database.

[1997] Output: A list of identified conference participants.

[1998] Step 3: Generate a participant list

[1999] Input: A list of identified conference participants.

[2000] Processing: The server generates this list and sends it to the terminal.

[2001] Output: Participant list sent to the device.

[2002] Step 4: View the Participant List

[2003] Input: The participant list sent to the device.

[2004] Processing: The terminal displays the received participant list on the conference screen. For example, names such as "Taro Tanaka" and "Hanako Sato" are displayed in the list.

[2005] Output: Participant list displayed on the conference screen.

[2006] Step 5: Capture audio

[2007] Input: User's spoken question.

[2008] Processing: During a meeting, a user asks a question by voice, such as "Where are the marketing materials?" The device captures the voice with its microphone and sends it to the server as audio data.

[2009] Output: The audio data sent to the server.

[2010] Step 6: Speech to Text

[2011] Input: The audio data sent to the server.

[2012] Processing: The server uses Google Cloud Speech-to-Text to convert the audio data into text. For example, the audio is converted into text, such as "Where are the marketing materials?"

[2013] Output: The question converted to text.

[2014] Step 7: Parsing the Question

[2015] Input: The question converted to text.

[2016] Processing: The server analyzes the textual question and searches the database for relevant information and materials.

[2017] Output: Search results for related materials.

[2018] Step 8: Find and generate relevant materials

[2019] Input: Search results for related materials.

[2020] Processing: The server generates the necessary materials based on the search results, for example preparing a PDF file of marketing materials.

[2021] Output: The generated material.

[2022] Step 9: Viewing the material

[2023] Input: Generated materials.

[2024] Processing: The server sends the generated materials to the terminal, which displays them on the conference screen. The user can view the displayed materials.

[2025] Output: Materials displayed on the meeting screen.

[2026] Step 10: Capturing Emotional Data

[2027] Input: Facial expressions and tone of voice of meeting participants.

[2028] Processing: The device uses a camera and microphone to capture emotional data such as the user's facial expressions and tone of voice in real time.

[2029] Output: Captured emotion data.

[2030] Step 11: Analyze the emotion data

[2031] Input: Captured emotion data.

[2032] Processing: An emotion engine (e.g., Microsoft Azure Emotion API) analyzes the captured data and determines the user's emotion. For example, it outputs an analysis result such as "The user looks anxious."

[2033] Output: Parsed emotion data.

[2034] Step 12: Dynamic information provision and documentation updates

[2035] Input: Parsed emotion data.

[2036] Processing: The emotion engine sends the analysis results to the server. The server then provides dynamic information based on the emotion data, adapting to the situation. For example, if the user appears anxious, the server provides additional materials with additional explanations. The device then displays the latest materials on the conference screen.

[2037] Output: Updated documentation and information.

[2038] Step 13: Data collection and analysis

[2039] Input: Audio, text, materials, emotional data, etc. accumulated during the meeting.

[2040] Processing: The server centralizes this data and performs detailed analysis.

[2041] Output: Analysis results.

[2042] Step 14: Categorize and analyze the issue

[2043] Input: Analysis results.

[2044] Processing: Based on the analysis results, the server categorizes the issues discussed and conducts a detailed analysis.

[2045] Output: A list of issues sorted by category.

[2046] Step 15: Generate and share your report

[2047] Input: A list of issues sorted by category.

[2048] Processing: The server generates a report that describes the solutions and prediction results for each problem. The report is sent to the terminal and can be viewed by the meeting participants.

[2049] Output: Report shared with meeting participants.

[2050] (Application example 2)

[2051] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2052] Modern factories are required to immediately solve problems faced by workers and improve productivity. However, conventional systems lack real-time problem-solving capabilities and support that adapts to the emotions of workers. As a result, work efficiency declines and work errors become more likely to occur. The present invention aims to provide a real-time problem-solving assistant system specialized for factory environments to solve these problems.

[2053] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for identifying the faces of factory workers, means for generating a list of identified factory workers, means for converting the contents of questions from factory workers into text in real time using voice recognition technology, means for searching and generating procedures and information related to the contents of the questions, and means for classifying problems that occur during factory work into categories and presenting analysis results and solutions. This makes it possible to quickly solve problems faced by factory workers and improve work efficiency.

[2054] "Conference participant" refers to any person participating in a conference.

[2055] "Means for identifying faces" refers to technology that uses a camera or facial recognition algorithm to recognize the face of a target person and identify that person.

[2056] The "means for generating a list of identified conference participants" is a technology for creating a list of participants based on information about conference participants obtained by face recognition.

[2057] "Speech recognition technology" is a technology that converts human speech into text data in real time.

[2058] "Means for searching and generating materials and information" refers to technology that searches for the necessary materials and information from related databases and information sources based on the content of a user's question and provides it.

[2059] "Means of categorizing issues and presenting analysis results and solutions" is a technique for organizing and classifying problems and issues discussed during meetings or work, and proposing solutions to them.

[2060] "Factory worker" refers to a worker who is in charge of work in a factory.

[2061] "Means for identifying the faces of factory workers" refers to technology that uses cameras and facial recognition algorithms to recognize the faces of target workers and identify them.

[2062] The "means for generating a list of identified factory workers" is a technology for creating a list of on-site workers based on information about factory workers obtained through facial recognition.

[2063] "Means for converting the content of questions posed by factory workers into text in real time using voice recognition technology" refers to technology for converting the voices spoken by factory workers into text in real time.

[2064] The "means for searching and generating procedures and information related to the content of a question" is a technology that searches a database for and provides relevant work procedures and information based on the content of a question posed by a factory worker.

[2065] "Means of categorizing problems that occur during factory work and presenting analysis results and solutions" is a technique for organizing and classifying troubles and problems that occur during work and proposing solutions to them.

[2066] This invention is a real-time problem-solving assistant system for improving work efficiency in a factory environment. This system mainly consists of a server, a terminal, a user, and an emotion engine.

[2067] Server Features

[2068] The server is the center of the system and has the following important functions:

[2069] Facial recognition: Identify the faces of factory workers from camera footage and compare them with a pre-registered database. Uses the dlib library and OpenCV as facial recognition algorithms.

[2070] Question content analysis: The questions from the workers are converted into text using speech recognition technology, and the content is analyzed. This is done using the speech_recognition library.

[2071] Document search: Search and generate the necessary procedures and information from a database that stores work procedures and equipment information within the factory.

[2072] Problem analysis: Analyze the data accumulated during the work and classify and analyze the problems that arise by category.

[2073] Solution Proposal: Based on the analysis results, we provide future solutions and predicted results.

[2074] Emotion data analysis: Receive data from the emotion engine and use it for follow-up during and after work.

[2075] Device Features

[2076] The terminal is an input device used by factory workers and has the following functions:

[2077] Camera and microphone: Captures video and audio of factory workers and sends them to a server.

[2078] Display device: The procedure and information sent from the server are displayed on the work screen. Display devices include smart glasses and smartphones.

[2079] Emotion Recognition: Analyzes the facial expressions and tone of voice of workers to obtain emotional data. Emotion recognition is performed using the Affectiva API.

[2080] User Actions

[2081] The user refers to a factory worker and performs the following actions:

[2082] Voice questions: Ask questions by voice as you work.

[2083] View Instructions: View instructions and information displayed on your device.

[2084] Emotion Engine Functions

[2085] The emotion engine is a component that recognizes and analyzes the emotions of workers.

[2086] Emotion analysis: Analyze workers' emotions in real time through facial expression recognition and voice analysis.

[2087] Data provision: The analysis results are sent to the server and used to dynamically change procedures and information.

[2088] Specific examples

[2089] For example, consider a situation where a problem occurs during the operation of a machine in a factory.

[2090] Step 1: When work begins, the device captures the camera image, and the server identifies the worker. A list of workers is displayed.

[2091] Step 2: When the worker asks, "Why is the machine stuck?" the device captures the voice, the server transcribes it, and generates the relevant repair instructions.

[2092] Step 3: If the emotion engine senses confusion or anxiety from the worker during the work, the server updates the displayed repair procedure with additional information to support the worker.

[2093] Step 4: After the work is completed, the server analyzes the accumulated data and generates a report that presents solutions such as "machine maintenance methods" and "future improvements," which is shared with the worker via their terminal.

[2094] Prompt Sentence Examples

[2095] "Devant a system that describes the situation when a maintenance issue occurs in the factory and generates real-time solutions to specific qu...

Claims

1. means for identifying faces of conference participants; means for generating a list of identified conference participants; A means of converting questions from conference participants into text in real time using voice recognition technology; A means for searching and generating materials and information related to the question; A means to categorize the issues discussed during the meeting and present the analysis and solutions. A system including:

2. The system according to claim 1 , further comprising means for analyzing the contents discussed after the meeting and generating a prediction result.

3. 10. The system of claim 1, further comprising means for applying a facial recognition algorithm to identify conference participants.

Images