System

Abstract

A system is provided.SOLUTION: This system is provided with a means for inputting the data of moving images, voices, images and texts, a means for analyzing the inputted data and extracting character information, a means for automatically generating materials by using a template defined on the basis of the extracted character information and a means for preserving and distributing the generated materials.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Creating documents for daily work requires a great deal of time and effort, which causes stress for many people. Documents such as handover documents, knowledge sharing materials, meeting minutes, business trip reports, and interview and event articles are particularly difficult to create manually because they contain a mixture of data formats (video, audio, images, and text). This reduces work efficiency and increases the risk of important information being overlooked or misunderstood. A system that can solve these problems and reduce the burden of document creation is needed. [Means for solving the problem]

[0005] To solve this problem, the present invention provides the following means. First, it provides a means for users to input video, audio, image, and text data. Next, it includes a means for analyzing the input data on a server, converting video and audio data into text using voice recognition technology, and extracting text information from image data using optical character recognition technology. It also includes a means for automatically generating materials using templates defined based on the extracted text information. Finally, it provides a system that includes a means for saving the generated materials and distributing them to users. This can significantly reduce the time and effort required to create materials and dramatically improve business efficiency.

[0006] "Video" is a media file that contains video and audio, and is a file format that conveys information through both sight and sound.

[0007] "Audio" is sound waves converted into digital data, and is a file format that contains auditory information such as spoken words and environmental sounds.

[0008] "Image" refers to still image data that records visual information, and includes visual representations such as photographs and illustrations.

[0009] "Text" is digital data of information expressed using letters and symbols, and provides information in a written format such as a document or memo.

[0010] "Analysis" is the process of examining and examining input data using various techniques to extract or convert specific information.

[0011] "Speech recognition technology" is a technology that analyzes voice data and converts it into text information, automatically converting human speech into text.

[0012] "Optical character recognition technology" is a technology that analyzes character information contained in image data and converts it into text, extracting character information from printed or handwritten documents.

[0013] A "template" is a standard format for creating documents, a document template that specifies a specific format and layout.

[0014] "Materials" are documents that summarize the contents of business or meetings, and are documents that summarize information for a specific purpose.

[0015] "Automatic generation" is the process by which a system follows pre-programmed instructions to create materials without user intervention. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention is a system that automatically generates materials based on video, audio, images, and text provided by users. This system analyzes input data and automatically creates materials according to defined templates, significantly reducing the effort required for creating materials. Below, the program processing of this system is explained in natural language, and an embodiment is shown with specific examples.

[0038] System Overview

[0039] The system consists of the following major components:

[0040] 1. Data Entry Component

[0041] It provides an interface that allows users to input the content they want to document in the form of video, audio, images, and text.

[0042] Specifically, users log in to the system's portal and upload data.

[0043] 2. Data Analysis Component

[0044] The server analyzes the uploaded data, converting audio and video data into text using voice recognition technology, and extracting text information from image data using optical character recognition technology.

[0045] After analysis, the obtained text data is organized and integrated into a single text document.

[0046] 3. Document Generation Component

[0047] The server automatically generates materials based on the analyzed text data according to predefined templates.

[0048] For example, format it into meeting minutes, business trip reports, knowledge sharing materials, etc.

[0049] 4. Material Storage and Distribution Component

[0050] The server stores the generated materials in the user's personal or project folder for easy access by the user.

[0051] Users can download the generated materials from within the system and share them with relevant parties as needed.

[0052] Specific processing flow

[0053] Data entry examples

[0054] For example, suppose a user wants to upload a video of a presentation they gave on a business trip. After logging in to the system, the user selects the video file and clicks the upload button. At this time, the device sends the video file to the server and notifies the user when the upload is complete.

[0055] Specific examples of data analysis

[0056] When a video file is uploaded to the server, the server converts the audio in the video into text using speech recognition technology. If the audio data is in English, a speech recognition engine for the corresponding language is used to classify and convert the spoken content for each speaker. In addition, optical character recognition technology is used to extract text information from slide images that may be included in the video.

[0057] Example of data generation

[0058] The analyzed text data is organized and arranged according to a predefined business trip report template. The server automatically generates a report that highlights the presentation's main points and conclusions. If necessary, it also adds metadata such as the presentation date and trip destination.

[0059] Examples of material preservation and distribution

[0060] The server saves the automatically generated trip report in the user's personal folder. Once the save is complete, the user is notified and can download the report by logging back into the system. The downloaded report can be distributed to relevant parties via email or the system's sharing function.

[0061] In this way, the present invention automates each process of creating materials, thereby reducing the burden on users and significantly improving work efficiency.

[0062] The processing flow will be explained below.

[0063] Step 1:

[0064] The user logs into the system's captive portal.

[0065] Step 2:

[0066] The user selects video, audio, image, or text data and clicks the upload button.

[0067] Step 3:

[0068] The terminal transmits the selected file to the server.

[0069] Step 4:

[0070] The server receives the uploaded data and identifies the data type (video, audio, image, text).

[0071] Step 5:

[0072] The server applies automatic speech recognition (ASR) technology to the audio and video files, converting the audio data into text.

[0073] Step 6:

[0074] The server applies optical character recognition (OCR) technology to the image file to extract text information from the image.

[0075] Step 7:

[0076] The server analyzes the text data and extracts sentence structure, keywords, and important points.

[0077] Step 8:

[0078] The server consolidates all the analyzed text data and organizes it into a single document.

[0079] Step 9:

[0080] The server automatically generates materials using the integrated text data according to a predefined template.

[0081] Step 10:

[0082] The server stores the generated materials in the user's personal or project folder.

[0083] Step 11:

[0084] The server notifies the user when the document is complete.

[0085] Step 12:

[0086] The user logs back into the system to view and download the completed document.

[0087] Step 13:

[0088] The user shares the generated materials with the relevant parties as needed.

[0089] Example 1

[0090] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0091] In today's business environment, there is a need to organize large amounts of data and create documents quickly and accurately. However, when dealing with data in various formats (video, audio, images, text), much of the processing is done manually, which requires time and effort, resulting in a decrease in work efficiency. Furthermore, errors are likely to occur during the data analysis and document creation process, resulting in inconsistent quality of the results. A system that can solve these problems and improve work efficiency and document quality is needed.

[0092] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0093] In this invention, the server includes: a means for a user to input data in the form of video, audio, image, or text; a means for the server to analyze the input data and extract character information using voice recognition technology and optical character recognition technology; a means for the server to automatically generate materials based on the extracted character information using a defined template; and a means for the server to store the generated materials and notify and distribute them to users. This makes it possible to effectively analyze and integrate various data formats, quickly and automatically generate high-quality materials, and significantly reduce the user's workload.

[0094] "User" means any person or entity that wishes to use the System to input data and generate materials.

[0095] "Server" refers to a computer system that receives data from users, analyzes it, and automatically generates materials.

[0096] "Data" means information, whether in the form of video, audio, images, or text, entered into the system by a user.

[0097] "Means for input" refers to the interface and functionality for users to upload data into the system.

[0098] "Analyzing" refers to the process in which the server uses voice recognition technology, optical character recognition technology, and the like to extract text information from the input data.

[0099] "Voice recognition technology" refers to the technology that analyzes voice data and converts the content into text format.

[0100] "Optical character recognition technology" refers to the technology that analyzes character information contained in image data and converts it into text format.

[0101] "Defined template" refers to a pre-set format or style used when automatically generating materials.

[0102] The "means for automatically generating materials" refers to a process in which the server creates materials using a template defined based on the extracted character information.

[0103] "Storing" refers to the server storing the generated material in its own storage device in order to provide it to the user.

[0104] "Distributing" refers to the server notifying the user of the generated material and making it accessible to the user.

[0105] This invention is a system that automatically generates materials based on video, audio, images, and text provided by users. The system consists of the following main components:

[0106] Data Entry Components

[0107] Users log in to the system's portal and use an interface to input the content they want to document in the form of video, audio, images, and text. Specifically, users select a file and click the upload button to send the data from their device to the server.

[0108] Data Analysis Components

[0109] The server analyzes the uploaded data. For example, for video and audio data, speech recognition technology (e.g., Google Speech-to-Text API) is used to convert the speech into text. For image data, optical character recognition technology (e.g., Google Cloud Vision API) is used to extract text information. The resulting text data is organized and integrated into a single text document.

[0110] Document generation component

[0111] The server automatically generates documents based on the analyzed text data according to predefined templates (e.g., Microsoft Word templates), for example, meeting minutes, business trip reports, knowledge sharing materials, etc.

[0112] Material storage and distribution component

[0113] The server saves the generated materials in the user's personal or project folder, and once the saving is complete, the server notifies the user and provides a download link for the materials, allowing the user to easily access the materials.

[0114] Specific processing flow example

[0115] 1. Example of data entry

[0116] If a user wants to upload a video of a presentation they gave on a business trip, they log in to the system, select "presentation.mp4," and click the upload button. The device sends this video file to the server, and the server sends a notification that it has been received. The user sees a message saying, "Upload completed."

[0117] 2. Specific examples of data analysis

[0118] The server analyzes "presentation.mp4" and converts the audio into text using the Google Speech-to-Text API. As a result of the analysis, text data such as "The main points of today's presentation are..." is extracted. Furthermore, text information such as "Figure 1: Sales Trends" is extracted from the slide images in the video using the Google Cloud Vision API, and all text data is integrated into a single document.

[0119] 3. Specific examples of data generation

[0120] The server generates a business trip report template based on the integrated text data. The content includes sections such as "Presentation Overview," "Main Points," and "Conclusion." Metadata such as "Date: October 10, 2023" and "Location: Tokyo" are added.

[0121] 4. Examples of material preservation and distribution

[0122] The generated business trip report is saved in the user's personal folder as "report20231010.docx." The server sends the user a notification that "The document has been generated and is available for download." The user then logs in to the system again, clicks the download link, and downloads the report. The downloaded report can be shared with team members via email.

[0123] Prompt Sentence Examples

[0124] For example, the following prompts can explain the process of uploading a video of a presentation a user gave on a business trip, converting the audio in the video into text using speech recognition technology, and automatically generating materials based on a template for a business trip report:

[0125] Simply upload a video of your business trip presentation to the system. We use speech recognition technology to convert the audio in the video into text, and automatically generate a business trip report template. The report will highlight key points and conclusions, and include metadata about the presentation date and the business trip destination.

[0126] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0127] Step 1: Data entry

[0128] Input: Users log in to the system's portal and select the video, audio, image, or text data they wish to document.

[0129] Specific operation: The user selects a file on the upload screen and clicks the upload button, at which point the device sends the selected file to the server.

[0130] Output: The server receives the file and sends a notification to the user that the upload is complete. The user sees the message "Upload completed."

[0131] Step 2: Receiving and storing data

[0132] Input: Files sent from the device (video, audio, image, text data).

[0133] Specific operation: The server stores the received file in temporary storage, which is used for subsequent analysis.

[0134] Output: The path and filename of the saved file. For example, / tmp / presentation.mp4.

[0135] Step 3: Data analysis (speech recognition)

[0136] Input: A video or audio file stored in temporary storage.

[0137] Specific operation: The server converts the voice data into text using speech recognition technology (e.g., Google Speech-to-Text API).

[0138] Data processing: Convert audio data in video or audio files into text format.

[0139] Output: The converted text data, e.g., "The main points of the presentation are...".

[0140] Step 4: Data Analysis (Optical Character Recognition)

[0141] Input: An image file or a frame from a video file saved in temporary storage.

[0142] Specific operation: The server uses optical character recognition technology (e.g., Google Cloud Vision API) to extract text information from the image.

[0143] Data processing: Converts text information in image files into text format.

[0144] Output: Extracted text data. For example, "Figure 1: Sales Trends."

[0145] Step 5: Data Integration

[0146] Input: Text data extracted using speech recognition and optical character recognition techniques.

[0147] Specific operation: The server organizes and integrates the multiple text data obtained into a single text document.

[0148] Data processing: Integrate scattered text data and organize it into documents.

[0149] Output: A consolidated text document, e.g., "Presentation Report Text Data".

[0150] Step 6: Generate materials

[0151] Input: A consolidated text document.

[0152] Specific operation: The server uses a predefined template to automatically generate documents based on the integrated text data, for example, using a Microsoft Word template.

[0153] Data processing: Place text documents according to templates and add required metadata (e.g. date, location, etc.).

[0154] Output: An automatically generated document file, for example, a business trip report (report20231010.docx).

[0155] Step 7: Preservation of materials and user notification

[0156] Input: Auto-generated documentation file.

[0157] Specific operation: The server saves the generated materials in the user's personal folder or project folder, and notifies the user when the saving is complete.

[0158] Output: The path and download link of the saved document, for example, / user / reports / report20231010.docx.

[0159] Step 8: Download and share materials

[0160] Input: The download link provided to the user.

[0161] Specific operation: The user logs in to the system again, clicks the download link to download the document, and then shares the document with relevant parties as needed.

[0162] Output: Downloaded documents and share them. For example, send a business trip report to a team member via email.

[0163] By following the above steps, this system automates the process from analyzing input data to generating, saving, and distributing materials, significantly reducing the user's workload.

[0164] (Application example 1)

[0165] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0166] Recording work processes and creating reports in factories requires a huge amount of time and effort when done manually, and there is a high risk of errors and information omissions. It also presents challenges in accurately understanding work content and quickly troubleshooting. Under these circumstances, automating work records and quickly generating reports is required to improve work efficiency and quality control throughout the factory.

[0167] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0168] In this invention, the server includes means for inputting video, audio, image, and text data, means for analyzing the input data and extracting character information, means for automatically generating materials using templates defined based on the extracted character information, means for saving and distributing the generated materials, and means for analyzing data captured by a factory work robot and automatically generating work reports. This automates the recording of work processes and the creation of reports in a factory, making it possible to improve work efficiency and strengthen quality control.

[0169] "Video" is data that records moving images or videos.

[0170] "Audio" is data that is a recording of a human voice or other sounds.

[0171] An "image" is data that records still visual information.

[0172] "Text" is information expressed in characters.

[0173] "Input means" refers to a method or device that allows a user or robot to provide video, audio, image, or text data to the system.

[0174] "Means of analysis" are techniques and methods for extracting useful information from the provided data.

[0175] "Text information" is information expressed as text.

[0176] "Extraction means" refers to the technology or equipment used to extract the necessary information from the data.

[0177] A "defined template" is a set of rules or formats for creating materials according to a specific format or structure.

[0178] "Means for automatically generating materials" refers to techniques or methods for automatically creating materials based on extracted information in accordance with predefined templates.

[0179] "Means for storing and distributing" refers to the techniques and methods for storing the automatically generated materials and distributing them to users who need them.

[0180] A "factory work robot" is a robotic device that automatically performs work in a factory.

[0181] A "work report" is a document that lists details of the work, results, problems, etc.

[0182] The following describes in detail an embodiment of the present invention. The present invention is a system that analyzes video, audio, and image data captured by a factory work robot and automatically generates a work report based on the data. The system of the present invention includes means for inputting video, audio, image, and text data, and means for analyzing each data and extracting text information. It also includes means for automatically generating materials using templates defined based on the extracted text information, and means for saving and distributing the generated materials. The specific configuration and operation of each means are described below.

[0183] First, the video, audio, and image data captured by the factory robot during work is sent to the system's server via input means, which include the robot's camera, microphone, and image capture device. The user then gives the robot appropriate instructions to collect data.

[0184] The server then analyzes the video, audio, and image data and extracts the necessary text information. This analysis uses the following techniques:

[0185] For voice recognition, a voice recognition engine (e.g., "Google Cloud Speech-to-Text") is used.

[0186] To extract text information from image data, an optical character recognition engine (e.g., "Tesseract OCR") is used.

[0187] A video analysis engine (e.g., OpenCV and deep learning models) is used to analyze the movements.

[0188] After data analysis, the extracted text information is automatically organized and consolidated by the server based on a predefined work report template, which includes, for example, the steps performed by the robot, any problems detected, and any suggested improvements.

[0189] Finally, the generated work report is stored by the server and distributed to the relevant parties as needed. The authenticity of the stored report may be verified by a digital signature, ensuring the reliability of the report content. Through this storage and distribution method, users and administrators can easily access and share the report.

[0190] A concrete example of this would be the following scenario. Suppose a robot is inspecting products on a factory production line. The robot captures video and audio during the inspection and sends the data to a server. The server analyzes this data and automatically generates a report containing the inspection results and any detected defects. This report becomes an important source of information for managers to view and take any necessary corrective measures promptly.

[0191] Examples of prompts using generative AI models include:

[0192] Generative AI prompts for work report generation systems:

[0193] 1. Please generate a work report based on the following text data.

[0194] 2. It should include a summary of the work, any issues found, and any suggested improvements.

[0195] 3. The output format is a Microsoft Word document.

[0196] In this way, the system of the present invention contributes to efficient management of factory work and improved quality.

[0197] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0198] Step 1:

[0199] Data collection by factory robots

[0200] input:

[0201] The user gives instructions to the factory robot to start work, and the robot collects video, audio, and image data using a camera, microphone, and image capture device.

[0202] output:

[0203] Data files for video (mp4 format), audio (wav format), and images (png or jpg format) are generated.

[0204] Specific behavior:

[0205] The robot's camera captures the work scene.

[0206] The microphone records your voice as you work.

[0207] An image capture device periodically takes still images.

[0208] Step 2:

[0209] Uploading data

[0210] input:

[0211] The video, audio, and image data collected by the robot is managed on a terminal.

[0212] output:

[0213] The data file is uploaded to the server.

[0214] Specific behavior:

[0215] The terminal receives data from the robot.

[0216] The device uploads the data to a server via the Internet.

[0217] Step 3:

[0218] Data analysis

[0219] input:

[0220] Video, audio, and image data uploaded to the server.

[0221] output:

[0222] Text data of character information and action information analyzed by the server.

[0223] Specific behavior:

[0224] The server analyzes the video data and extracts motion information for each specified frame.

[0225] The voice data is converted into text using a voice recognition engine (Google Cloud Speech-to-Text).

[0226] Text information is extracted from the image data using an optical character recognition engine (Tesseract OCR).

[0227] Step 4:

[0228] Material generation

[0229] input:

[0230] Text data of character information and action information analyzed by the server.

[0231] output:

[0232] Automatically generated work reports (Microsoft Word documents) based on predefined templates.

[0233] Specific behavior:

[0234] The server arranges the extracted text information according to a defined template.

[0235] Automatically generate work reports that include work details, issues detected, and suggested improvements.

[0236] Step 5:

[0237] Storage and distribution of materials

[0238] input:

[0239] Automatically generated work report (Microsoft Word document).

[0240] output:

[0241] Work reports stored in the server's cloud storage and distributed to stakeholders.

[0242] Specific behavior:

[0243] The server stores the generated work report in cloud storage.

[0244] Users can access the saved reports and distribute them to interested parties as needed.

[0245] Digital signature functionality is used to ensure the authenticity of reports.

[0246] The above steps create a system that automatically records factory work and creates reports, which improves work efficiency and strengthens quality control.

[0247] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0248] This invention combines a system that automatically generates materials based on user-provided video, audio, images, and text with an emotion engine that recognizes the user's emotions, enabling it to adjust the tone and content of the materials according to the user's emotional state. This system analyzes input data, extracts the user's emotional information, and automatically creates materials according to defined templates, significantly reducing the effort required to create materials and providing materials with more appropriate content.

[0249] System Overview

[0250] The system consists of the following major components:

[0251] 1. Data Entry Component

[0252] It provides an interface that allows users to input the content they want to document in the form of video, audio, images, and text.

[0253] Users log in to the system's portal and upload data.

[0254] 2. Data Analysis Component

[0255] The server analyzes the uploaded data, converting audio and video data into text using voice recognition technology, and extracting text information from image data using optical character recognition technology.

[0256] An emotion engine is used to recognize the user's emotions from video and audio, and organizes the emotional information together with text data.

[0257] 3. Document Generation Component

[0258] The server automatically generates materials based on the analyzed text data and emotional information according to predefined templates.

[0259] Adjust the tone and content of your materials depending on the user's emotional state.

[0260] 4. Material Storage and Distribution Component

[0261] The server stores the generated materials in the user's personal or project folder for easy access by the user.

[0262] Users can download the generated materials from within the system and share them with relevant parties as needed.

[0263] Specific processing flow

[0264] Data entry examples

[0265] For example, consider a case where a user wants to upload a recording of a meeting to the system. The user logs in to the system, selects the recording file, and clicks the upload button. The device then sends the recording file to the server and notifies the user when the upload is complete.

[0266] Specific examples of data analysis

[0267] Once the recording file is uploaded to the server, the server converts the voice data into text using speech recognition technology. At the same time, the emotion engine analyzes the tone, pitch, and speed of the voice to recognize the user's emotions. For example, if someone expresses strong interest or dissatisfaction during a meeting, that emotional information is recorded along with the text data.

[0268] Example of data generation

[0269] The analyzed text data and emotional information are organized and arranged according to a predefined meeting minutes template. The server automatically generates minutes based on the emotional information, emphasizing specific remarks and discussion points in an appropriate tone. For example, points that users showed strong interest in are recorded in particular detail.

[0270] Examples of material preservation and distribution

[0271] The automatically generated meeting minutes are saved in the user's personal folder by the server. Once saved, the user is notified and can download the minutes by logging back into the system. The downloaded minutes can then be distributed to relevant parties via email or the system's sharing function.

[0272] In this way, the present invention automates each process of document creation and further utilizes user emotional information to generate more accurate and appropriate documents, significantly improving work efficiency.

[0273] The processing flow will be explained below.

[0274] Step 1:

[0275] The user logs into the system's captive portal.

[0276] Step 2:

[0277] The user selects video, audio, image, or text data and clicks the upload button.

[0278] Step 3:

[0279] The terminal transmits the selected file to the server.

[0280] Step 4:

[0281] The server receives the uploaded data and identifies the data type (video, audio, image, text).

[0282] Step 5:

[0283] The server applies automatic speech recognition (ASR) technology to the audio and video files, converting the audio data into text.

[0284] Step 6:

[0285] The server applies optical character recognition (OCR) technology to the image file to extract text information from the image.

[0286] Step 7:

[0287] The server applies an emotion engine to the video or audio file to recognize the user's emotion. For example, the emotion engine analyzes the tone, pitch, and speed of the voice to identify whether the user is happy, angry, or surprised.

[0288] Step 8:

[0289] The server analyzes the text data and emotional information to extract sentence structure, keywords, and key points. The emotional information is used to emphasize and refine specific parts.

[0290] Step 9:

[0291] The server integrates all the analyzed text data and emotional information and organizes it into a single document.

[0292] Step 10:

[0293] The server automatically generates materials using the integrated text data and emotional information according to predefined templates, adjusting the tone and content of the materials by, for example, highlighting parts that express strong emotions.

[0294] Step 11:

[0295] The server stores the generated materials in the user's personal or project folder.

[0296] Step 12:

[0297] The server notifies the user when the document is complete.

[0298] Step 13:

[0299] The user logs back into the system to view and download the completed document.

[0300] Step 14:

[0301] Users can share the generated materials with stakeholders as needed by creating a sharing link within the system or sending it via email.

[0302] Example 2

[0303] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0304] Conventional document creation systems have been unable to generate documents that take the user's emotions into account, and as a result, the automatically generated documents often do not match the user's intentions or emotions. In addition, because many parts of the system rely on manual work, document creation takes time and effort.

[0305] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for a user to input video, audio, still image, and document data, means for analyzing the input data and extracting text information, means for recognizing the user's emotional information and adjusting the tone and content of the materials based on the emotional information, means for automatically generating materials using templates defined based on the extracted text information and emotional information, and means for saving and distributing the generated materials. This makes it possible to generate highly accurate materials that reflect the user's emotions.

[0306] "User" refers to any individual or organization that uses the System to input data and generate or download materials.

[0307] "Video" refers to digital or analog data that records moving images.

[0308] "Audio" refers to digital or analog data that records human speech or other sounds.

[0309] A "still image" refers to a single piece of image data that does not move.

[0310] "Document data" refers to data in the form of text files or documents that contain character information.

[0311] A "server" refers to a computer system that performs processes such as data analysis and automatic generation of materials.

[0312] "Template" refers to a defined format or layout used to automatically generate materials.

[0313] "Voice recognition technology" refers to the technology that analyzes voice data and converts it into text data.

[0314] "Optical character recognition technology (OCR)" refers to the technology of extracting character information from image data.

[0315] "Emotional information" refers to data obtained by analyzing the user's emotional state.

[0316] The "tone of a document" refers to the atmosphere and tone of the content and presentation of the document.

[0317] "Document generation" refers to the process of creating documents based on analyzed data.

[0318] "Preservation" refers to the keeping of generated materials in appropriate storage.

[0319] "Distribution" refers to providing generated materials to other users or interested parties.

[0320] This invention combines a system that automatically generates materials based on video, audio, still images, and text data provided by users with an emotion engine that recognizes the user's emotions, giving it the ability to adjust the tone and content of the materials according to the user's emotional state. This system analyzes the input data, extracts the user's emotional information, and automatically creates materials according to defined templates, significantly reducing the effort required to create materials and providing materials with more appropriate content.

[0321] System configuration

[0322] The system consists of the following major components:

[0323] 1. Data Entry Component

[0324] It provides an interface that allows users to input the content they want to document in the form of video, audio, still images, or documents.

[0325] Users log in to the system's portal and upload data.

[0326] 2. Data Analysis Component

[0327] The server analyzes the uploaded data, converting audio and video data into text using voice recognition technology, and extracting text information from still image data using optical character recognition (OCR).

[0328] An emotion engine is used to recognize the user's emotions from video and audio, and organizes the emotional information together with text data.

[0329] 3. Document Generation Component

[0330] The server automatically generates materials based on the analyzed text data and emotional information according to predefined templates.

[0331] Adjust the tone and content of your materials depending on the user's emotional state.

[0332] 4. Material Storage and Distribution Component

[0333] The server stores the generated materials in the user's personal or project folder for easy access by the user.

[0334] Users can download the generated materials from within the system and share them with relevant parties as needed.

[0335] Detailed component description

[0336] 1. Data Entry Component

[0337] Users first log in to the system's portal and upload the data they want to document (video, audio, still images, documents). The device then sends the selected files to the server and notifies the user when the upload is complete.

[0338] For example, if a user wants to upload a recording file of a meeting to the system, the user logs in, selects the recording file, and clicks the upload button. The device sends the recording file to the server and notifies the user when the upload is complete.

[0339] Prompt statement example (data entry):

[0340] A user logs in to the system, clicks the "Upload Data" button on the portal to upload the recording file of the meeting, selects the recording file and uploads it.

[0341] 2. Data Analysis Component

[0342] The server analyzes the uploaded data and uses speech recognition technology (e.g., speech recognition API) to convert audio and video data into text, and optical character recognition (OCR) technology (e.g., OCR software) to extract text information from still image data.

[0343] The server uses an emotion engine (e.g., emotion analysis API) to recognize the user's emotions from video and audio data, and organizes the emotional information together with the text data.

[0344] For example, after the server receives the uploaded recording file, it calls a speech recognition API to convert the voice data into text. At the same time, it uses an emotion analysis API to analyze the tone, pitch, speed, etc. of the voice to determine the user's emotion. For example, it generates a result such as "The user is angry" and stores it in a database along with the text data.

[0345] Example prompt (Data Analysis):

[0346] The server uses a speech recognition API to convert the voice data into text and an emotion analysis API to analyze the user's emotions.

[0347] 3. Document Generation Component

[0348] The server automatically generates materials based on the analyzed text data and emotion information according to a predefined template (e.g., document template).

[0349] The server generates materials based on the emotional information, emphasizing specific statements and discussion points in an appropriate tone. For example, parts that the user expressed strong interest in will be recorded in particular detail.

[0350] For example, the server reads the parsed text data and sentiment information, inserts these data into a template defined using the document template API, and the generated document is automatically saved as a "meeting minutes" file.

[0351] Example prompt (data generation):

[0352] The server organizes the analyzed text data and emotional information based on the document template API and automatically generates meeting minutes.

[0353] 4. Material Storage and Distribution Component

[0354] The server saves the generated materials in the user's personal folder or project folder (e.g., cloud storage service). Once the saving is complete, the user is notified.

[0355] Users can re-login to the system, access their personal folders, and download documents, and also share documents with relevant parties.

[0356] As a concrete example, the server uses a cloud storage API to save the generated "meeting minutes" file in the user's personal folder. After saving is complete, the server sends a notification to the user's device. The user then logs in again and downloads the document from their personal folder. The document can also be shared by clicking the "Share" button and entering the email addresses of those involved.

[0357] Example prompt (storage and distribution):

[0358] The server saves the automatically generated meeting minutes in cloud storage and notifies the user after saving is complete.

[0359] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0360] Step 1: Data entry

[0361] Specific explanation

[0362] Users log in to the system's portal and upload the data they want to document (video, audio, still images, documents). The terminal sends the selected files to the server and notifies the user when the upload is complete. The input is the file specified by the user, and the output is the data stored on the server.

[0363] Specific actions

[0364] After logging in, users can click the "Select File" button in the upload interface to select the file they want to document. By pressing the "Start Upload" button, the device will send the file to the server and display a progress bar. After the upload is complete, the device will display a notification message saying "Upload Completed."

[0365] Step 2: Data analysis

[0366] Specific explanation

[0367] The server analyzes the uploaded data. It uses voice recognition technology (e.g., voice recognition API) to convert audio and video data into text, and optical character recognition (OCR) technology (e.g., OCR software) to extract text information from still image data. It then uses an emotion engine (e.g., emotion analysis API) to recognize the user's emotions from the video and audio, and organizes the emotional information along with the text data. The input is the data stored on the server, and the output is the analyzed text data and emotional information.

[0368] Specific actions

[0369] After the server receives the uploaded audio file, it calls a speech recognition API to convert the audio data into text. At the same time, it uses an emotion analysis API to analyze the tone, pitch, speed, etc. of the voice to determine the user's emotion. For example, it generates a result such as "The user is angry" and stores it in the database along with the text data. For image files, it uses OCR software to extract characters and stores the analysis results in the database as text data.

[0370] Adding Inputs and Outputs

[0371] The input (data stored on the server) is audio files, image files, and document files, and the output (internal server data) is text data and emotional information obtained through analysis.

[0372] Step 3: Generate materials

[0373] Specific explanation

[0374] The server automatically generates materials based on the analyzed text data and emotional information according to a predefined template (e.g., a document template). The content of the materials is adjusted according to the user's emotional state. The input is the text data and emotional information stored on the server, and the output is the generated materials.

[0375] Specific actions

[0376] The server reads the parsed text data and emotion information and inserts this data into a template defined using the document template API. For example, it increases the font size or changes the color to highlight parts where the user expressed strong interest or anger. The generated document is saved as a "meeting minutes" file.

[0377] Adding Inputs and Outputs

[0378] The input (internal data of the server) is analyzed text data and emotion information, and the output (generated file) is material generated based on a template.

[0379] Step 4: Preservation and distribution

[0380] Specific explanation

[0381] The server saves the generated materials in the user's personal folder or project folder. Once saving is complete, a notification is sent to the user, who can then re-login to the system to download the materials. It also provides a function to share the materials with relevant parties. The input is the generated materials, and the output is the URL of the saved materials or a notification that saving has been completed.

[0382] Specific actions

[0383] The server uses the cloud storage API to save the generated "meeting minutes" file in the user's personal folder. After saving is complete, the server sends a notification to the user's device. The user can also log in to the system again, access their personal folder, and download the materials. The user can share the materials by clicking the "Share" button and entering the email addresses of the relevant parties.

[0384] Adding Inputs and Outputs

[0385] The input (generated file) is the generated material, and the output (user-accessible link or distributable data) is the URL of the material stored in cloud storage or a notification that the storage is complete.

[0386] (Application example 2)

[0387] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0388] Conventional document generation systems were capable of extracting textual information from user input data, but were unable to generate documents that reflected emotional information. As a result, users' intentions and nuances were sometimes not accurately conveyed, making it difficult to create more effective advertising proposals and documents. In particular, in marketing and advertising campaigns, proposals that capture the emotions of target users are required, so solving this problem is an urgent task.

[0389] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0390] In this invention, the server includes means for inputting video, audio, image, and text data, means for analyzing the input data and extracting character information and emotional information, means for automatically generating materials using templates defined based on the extracted character information and emotional information, and means for adjusting the tone and content of the materials based on the emotional information, and means for saving and distributing the generated materials. This enables the creation of more effective and detailed advertising proposal materials and materials that reflect the user's emotions.

[0391] "Motion picture" is a data format in which visual content is displayed continuously.

[0392] "Audio" is a data format that includes auditory information such as speech and sound effects.

[0393] An "image" is a data format for displaying fixed visual content.

[0394] "Text" is a data format that includes characters such as alphabets and kanji.

[0395] "Emotion information" is data that indicates the user's feelings and emotions, and is analyzed from the input data.

[0396] A "template" is a format used as a standard when generating materials.

[0397] "Materials" refers to documents or presentations created for a specific purpose or to organize information.

[0398] "Tone" refers to the overall atmosphere or mood of a document or piece of writing.

[0399] "Storage" refers to the act of recording generated materials in a database or file system.

[0400] "Distribution" refers to the act of sending or making available generated materials to other users or systems.

[0401] The present invention relates to a system for automatically generating proposal materials for an advertising campaign based on various input data. Hereinafter, an embodiment of the present invention will be described in detail.

[0402] First, the system provides an interface for inputting video, audio, image, and text data. Users log in to the system and upload this data, which is then sent to the server as the basis for creating materials. The server analyzes the input data and extracts text and emotional information. Audio data is converted into text using voice recognition technology, and image data is used to extract text information using optical character recognition technology.

[0403] At this time, emotional information is extracted from the analyzed audio and video using an emotion engine. This emotion engine analyzes the user's voice characteristics, such as tone, pitch, and speed, to recognize their emotional state. This emotional information and text information are then organized by the server.

[0404] The server then automatically generates ad campaign proposals using predefined templates based on the organized text and emotional information. The tone and content of the proposals are adjusted based on the user's emotional state. For example, specific ad copy or visuals may be emphasized based on areas of high interest.

[0405] The generated materials are stored on the server and made easily accessible to users, who can also download the materials from within the system and share them with interested parties.

[0406] The main hardware and software used in implementing the system are as follows:

[0407] Hardware: PC or smartphone with internet connection

[0408] Software: Python, cv2 library, speech_recognition library, transformers library

[0409] As a concrete example, consider a scenario in which a user wants to upload a recording of a meeting to a system. The user logs in to the system, selects the recording file, and clicks the upload button. The device then sends the recording file to the server. Once the upload is complete, the system converts the recording file into text using speech recognition technology, and automatically generates proposal materials for an advertising campaign based on the text data and emotional information.

[0410] An example prompt might look like this:

[0411] "Create an advertising campaign pitch from these voice memos. The memos include product features and highlights."

[0412] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0413] Step 1:

[0414] The user logs in to the system. The user enters authentication information on the provided login screen and accesses the system. This operation allows the user to specify which project to upload data to.

[0415] Step 2:

[0416] The user uploads video, audio, image, or text data. The user selects the data file on the upload screen and clicks the send button. The input data is sent from the device to the server. The input data format includes mp4, wav, jpg, pdf, txt, etc.

[0417] Step 3:

[0418] The server receives the uploaded data. The received data is first saved in a temporary location before being analyzed. For example, if audio data is provided in wav format, it will be saved in the specified directory.

[0419] Step 4:

[0420] The server analyzes the data and extracts text and emotional information. Specifically, it converts audio data into text using speech recognition technology, and extracts text from image data using optical character recognition (OCR). It then uses an emotion engine to analyze the tone, pitch, and speed of the audio and video to recognize the emotional state. In this step, audio or image data is received as input, and text and emotional information are generated as output.

[0421] Step 5:

[0422] The server automatically generates materials using templates defined based on text and emotional information. The server arranges the necessary data appropriately according to the template and adjusts the tone and content based on emotional information. For example, particularly detailed explanations are added to parts that show strong user interest. In this step, organized text and emotional information are received as input, and advertising campaign proposal materials are generated as output.

[0423] Step 6:

[0424] The server saves and distributes the generated materials. The automatically generated materials are saved in the user's personal folder or project folder. After saving is complete, a notification is sent to the user, who can then re-login to the system and download the materials. Furthermore, the user can share the downloaded materials with relevant parties via email or the sharing function. In this step, the generated materials are received as input, and saving and distribution are performed as output.

[0425] In this way, efficient and effective material creation is achieved through the specific processing steps of the system that automatically generates advertising campaign proposal materials that reflect user emotional information.

[0426] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0427] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0428] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0429] [Second embodiment]

[0430] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0431] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0432] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0433] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0434] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0435] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0436] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0437] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0438] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0439] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0440] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0441] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0442] The present invention is a system that automatically generates materials based on video, audio, images, and text provided by users. This system analyzes input data and automatically creates materials according to defined templates, significantly reducing the effort required for creating materials. Below, the program processing of this system is explained in natural language, and an embodiment is shown with specific examples.

[0443] System Overview

[0444] The system consists of the following major components:

[0445] 1. Data Entry Component

[0446] It provides an interface that allows users to input the content they want to document in the form of video, audio, images, and text.

[0447] Specifically, users log in to the system's portal and upload data.

[0448] 2. Data Analysis Component

[0449] The server analyzes the uploaded data, converting audio and video data into text using voice recognition technology, and extracting text information from image data using optical character recognition technology.

[0450] After analysis, the obtained text data is organized and integrated into a single text document.

[0451] 3. Document Generation Component

[0452] The server automatically generates materials based on the analyzed text data according to predefined templates.

[0453] For example, format it into meeting minutes, business trip reports, knowledge sharing materials, etc.

[0454] 4. Material Storage and Distribution Component

[0455] The server stores the generated materials in the user's personal or project folder for easy access by the user.

[0456] Users can download the generated materials from within the system and share them with relevant parties as needed.

[0457] Specific processing flow

[0458] Data entry examples

[0459] For example, suppose a user wants to upload a video of a presentation they gave on a business trip. After logging in to the system, the user selects the video file and clicks the upload button. At this time, the device sends the video file to the server and notifies the user when the upload is complete.

[0460] Specific examples of data analysis

[0461] When a video file is uploaded to the server, the server converts the audio in the video into text using speech recognition technology. If the audio data is in English, a speech recognition engine for the corresponding language is used to classify and convert the spoken content for each speaker. In addition, optical character recognition technology is used to extract text information from slide images that may be included in the video.

[0462] Example of data generation

[0463] The analyzed text data is organized and arranged according to a predefined business trip report template. The server automatically generates a report that highlights the presentation's main points and conclusions. If necessary, it also adds metadata such as the presentation date and trip destination.

[0464] Examples of material preservation and distribution

[0465] The server saves the automatically generated trip report in the user's personal folder. Once the save is complete, the user is notified and can download the report by logging back into the system. The downloaded report can be distributed to relevant parties via email or the system's sharing function.

[0466] In this way, the present invention automates each process of creating materials, thereby reducing the burden on users and significantly improving work efficiency.

[0467] The processing flow will be explained below.

[0468] Step 1:

[0469] The user logs into the system's captive portal.

[0470] Step 2:

[0471] The user selects video, audio, image, or text data and clicks the upload button.

[0472] Step 3:

[0473] The terminal transmits the selected file to the server.

[0474] Step 4:

[0475] The server receives the uploaded data and identifies the data type (video, audio, image, text).

[0476] Step 5:

[0477] The server applies automatic speech recognition (ASR) technology to the audio and video files, converting the audio data into text.

[0478] Step 6:

[0479] The server applies optical character recognition (OCR) technology to the image file to extract text information from the image.

[0480] Step 7:

[0481] The server analyzes the text data and extracts sentence structure, keywords, and important points.

[0482] Step 8:

[0483] The server consolidates all the analyzed text data and organizes it into a single document.

[0484] Step 9:

[0485] The server automatically generates materials using the integrated text data according to a predefined template.

[0486] Step 10:

[0487] The server stores the generated materials in the user's personal or project folder.

[0488] Step 11:

[0489] The server notifies the user when the document is complete.

[0490] Step 12:

[0491] The user logs back into the system to view and download the completed document.

[0492] Step 13:

[0493] The user shares the generated materials with the relevant parties as needed.

[0494] Example 1

[0495] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0496] In today's business environment, there is a need to organize large amounts of data and create documents quickly and accurately. However, when dealing with data in various formats (video, audio, images, text), much of the processing is done manually, which requires time and effort, resulting in a decrease in work efficiency. Furthermore, errors are likely to occur during the data analysis and document creation process, resulting in inconsistent quality of the results. A system that can solve these problems and improve work efficiency and document quality is needed.

[0497] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0498] In this invention, the server includes: a means for a user to input data in the form of video, audio, image, or text; a means for the server to analyze the input data and extract character information using voice recognition technology and optical character recognition technology; a means for the server to automatically generate materials based on the extracted character information using a defined template; and a means for the server to store the generated materials and notify and distribute them to users. This makes it possible to effectively analyze and integrate various data formats, quickly and automatically generate high-quality materials, and significantly reduce the user's workload.

[0499] "User" means any person or entity that wishes to use the System to input data and generate materials.

[0500] "Server" refers to a computer system that receives data from users, analyzes it, and automatically generates materials.

[0501] "Data" means information, whether in the form of video, audio, images, or text, entered into the system by a user.

[0502] "Means for input" refers to the interface and functionality for users to upload data into the system.

[0503] "Analyzing" refers to the process in which the server uses voice recognition technology, optical character recognition technology, and the like to extract text information from the input data.

[0504] "Voice recognition technology" refers to the technology that analyzes voice data and converts the content into text format.

[0505] "Optical character recognition technology" refers to the technology that analyzes character information contained in image data and converts it into text format.

[0506] "Defined template" refers to a pre-set format or style used when automatically generating materials.

[0507] The "means for automatically generating materials" refers to a process in which the server creates materials using a template defined based on the extracted character information.

[0508] "Storing" refers to the server storing the generated material in its own storage device in order to provide it to the user.

[0509] "Distributing" refers to the server notifying the user of the generated material and making it accessible to the user.

[0510] This invention is a system that automatically generates materials based on video, audio, images, and text provided by users. The system consists of the following main components:

[0511] Data Entry Components

[0512] Users log in to the system's portal and use an interface to input the content they want to document in the form of video, audio, images, and text. Specifically, users select a file and click the upload button to send the data from their device to the server.

[0513] Data Analysis Components

[0514] The server analyzes the uploaded data. For example, for video and audio data, speech recognition technology (e.g., Google Speech-to-Text API) is used to convert the speech into text. For image data, optical character recognition technology (e.g., Google Cloud Vision API) is used to extract text information. The resulting text data is organized and integrated into a single text document.

[0515] Document generation component

[0516] The server automatically generates documents based on the analyzed text data according to predefined templates (e.g., Microsoft Word templates), for example, meeting minutes, business trip reports, knowledge sharing materials, etc.

[0517] Material storage and distribution component

[0518] The server saves the generated materials in the user's personal or project folder, and once the saving is complete, the server notifies the user and provides a download link for the materials, allowing the user to easily access the materials.

[0519] Specific processing flow example

[0520] 1. Example of data entry

[0521] If a user wants to upload a video of a presentation they gave on a business trip, they log in to the system, select "presentation.mp4," and click the upload button. The device sends this video file to the server, and the server sends a notification that it has been received. The user sees a message saying, "Upload completed."

[0522] 2. Specific examples of data analysis

[0523] The server analyzes "presentation.mp4" and converts the audio into text using the Google Speech-to-Text API. As a result of the analysis, text data such as "The main points of today's presentation are..." is extracted. Furthermore, text information such as "Figure 1: Sales Trends" is extracted from the slide images in the video using the Google Cloud Vision API, and all text data is integrated into a single document.

[0524] 3. Specific examples of data generation

[0525] The server generates a business trip report template based on the integrated text data. The content includes sections such as "Presentation Overview," "Main Points," and "Conclusion." Metadata such as "Date: October 10, 2023" and "Location: Tokyo" are added.

[0526] 4. Examples of material preservation and distribution

[0527] The generated business trip report is saved in the user's personal folder as "report20231010.docx." The server sends the user a notification that "The document has been generated and is available for download." The user then logs in to the system again, clicks the download link, and downloads the report. The downloaded report can be shared with team members via email.

[0528] Prompt Sentence Examples

[0529] For example, the following prompts can explain the process of uploading a video of a presentation a user gave on a business trip, using speech recognition technology to convert the audio in the video into text, and automatically generating materials based on a template for a business trip report:

[0530] Simply upload a video of your business trip presentation to the system. We use speech recognition technology to convert the audio in the video into text, and automatically generate a business trip report template. The report will highlight key points and conclusions, and include metadata about the presentation date and the business trip destination.

[0531] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0532] Step 1: Data entry

[0533] Input: Users log in to the system's portal and select the video, audio, image, or text data they wish to document.

[0534] Specific operation: The user selects a file on the upload screen and clicks the upload button, at which point the device sends the selected file to the server.

[0535] Output: The server receives the file and sends a notification to the user that the upload is complete. The user sees the message "Upload completed."

[0536] Step 2: Receiving and storing data

[0537] Input: Files sent from the device (video, audio, image, text data).

[0538] Specific operation: The server stores the received file in temporary storage, which is used for subsequent analysis.

[0539] Output: The path and filename of the saved file. For example, / tmp / presentation.mp4.

[0540] Step 3: Data analysis (speech recognition)

[0541] Input: A video or audio file stored in temporary storage.

[0542] Specific operation: The server converts the voice data into text using speech recognition technology (e.g., Google Speech-to-Text API).

[0543] Data processing: Convert audio data in video or audio files into text format.

[0544] Output: The converted text data, e.g., "The main points of the presentation are...".

[0545] Step 4: Data Analysis (Optical Character Recognition)

[0546] Input: An image file or a frame from a video file saved in temporary storage.

[0547] Specific operation: The server uses optical character recognition technology (e.g., Google Cloud Vision API) to extract text information from the image.

[0548] Data processing: Converts text information in image files into text format.

[0549] Output: Extracted text data. For example, "Figure 1: Sales Trends."

[0550] Step 5: Data Integration

[0551] Input: Text data extracted using speech recognition and optical character recognition techniques.

[0552] Specific operation: The server organizes and integrates the multiple text data obtained into a single text document.

[0553] Data processing: Integrate scattered text data and organize it into documents.

[0554] Output: A consolidated text document, e.g., "Presentation Report Text Data".

[0555] Step 6: Generate materials

[0556] Input: A consolidated text document.

[0557] Specific operation: The server uses a predefined template to automatically generate documents based on the integrated text data, for example, using a Microsoft Word template.

[0558] Data processing: Place text documents according to templates and add required metadata (e.g. date, location, etc.).

[0559] Output: An automatically generated document file, for example, a business trip report (report20231010.docx).

[0560] Step 7: Preservation of materials and user notification

[0561] Input: Auto-generated documentation file.

[0562] Specific operation: The server saves the generated materials in the user's personal folder or project folder, and notifies the user when the saving is complete.

[0563] Output: The path and download link of the saved document, for example, / user / reports / report20231010.docx.

[0564] Step 8: Download and share materials

[0565] Input: The download link provided to the user.

[0566] Specific operation: The user logs in to the system again, clicks the download link to download the document, and then shares the document with relevant parties as needed.

[0567] Output: Downloaded documents and share them. For example, send a business trip report to a team member via email.

[0568] By following the above steps, this system automates the process from analyzing input data to generating, saving, and distributing materials, significantly reducing the user's workload.

[0569] (Application example 1)

[0570] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0571] Recording work processes and creating reports in factories requires a huge amount of time and effort when done manually, and there is a high risk of errors and information omissions. It also presents challenges in accurately understanding work content and quickly troubleshooting. Under these circumstances, automating work records and quickly generating reports is required to improve work efficiency and quality control throughout the factory.

[0572] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0573] In this invention, the server includes means for inputting video, audio, image, and text data, means for analyzing the input data and extracting character information, means for automatically generating materials using templates defined based on the extracted character information, means for saving and distributing the generated materials, and means for analyzing data captured by a factory work robot and automatically generating work reports. This automates the recording of work processes and the creation of reports in a factory, making it possible to improve work efficiency and strengthen quality control.

[0574] "Video" is data that records moving images or videos.

[0575] "Audio" is data that is a recording of a human voice or other sounds.

[0576] An "image" is data that records still visual information.

[0577] "Text" is information expressed in characters.

[0578] "Input means" refers to a method or device that allows a user or robot to provide video, audio, image, or text data to the system.

[0579] "Means of analysis" are techniques and methods for extracting useful information from the provided data.

[0580] "Text information" is information expressed as text.

[0581] "Extraction means" refers to the technology or equipment used to extract the necessary information from the data.

[0582] A "defined template" is a set of rules or formats for creating materials according to a specific format or structure.

[0583] "Means for automatically generating materials" refers to techniques or methods for automatically creating materials based on extracted information in accordance with predefined templates.

[0584] "Means for storing and distributing" refers to the techniques and methods for storing the automatically generated materials and distributing them to users who need them.

[0585] A "factory work robot" is a robotic device that automatically performs work in a factory.

[0586] A "work report" is a document that lists details of the work, results, problems, etc.

[0587] The following describes in detail an embodiment of the present invention. The present invention is a system that analyzes video, audio, and image data captured by a factory work robot and automatically generates a work report based on the data. The system of the present invention includes means for inputting video, audio, image, and text data, and means for analyzing each data and extracting text information. It also includes means for automatically generating materials using templates defined based on the extracted text information, and means for saving and distributing the generated materials. The specific configuration and operation of each means are described below.

[0588] First, the video, audio, and image data captured by the factory robot during work is sent to the system's server via input means, which include the robot's camera, microphone, and image capture device. The user then gives the robot appropriate instructions to collect data.

[0589] The server then analyzes the video, audio, and image data and extracts the necessary text information. This analysis uses the following techniques:

[0590] For voice recognition, a voice recognition engine (e.g., "Google Cloud Speech-to-Text") is used.

[0591] To extract text information from image data, an optical character recognition engine (e.g., "Tesseract OCR") is used.

[0592] A video analysis engine (e.g., OpenCV and deep learning models) is used to analyze the movements.

[0593] After data analysis, the extracted text information is automatically organized and consolidated by the server based on a predefined work report template, which includes, for example, the steps performed by the robot, any problems detected, and any suggested improvements.

[0594] Finally, the generated work report is stored by the server and distributed to the relevant parties as needed. The authenticity of the stored report may be verified by a digital signature, ensuring the reliability of the report content. Through this storage and distribution method, users and administrators can easily access and share the report.

[0595] A concrete example of this would be the following scenario. Suppose a robot is inspecting products on a factory production line. The robot captures video and audio during the inspection and sends the data to a server. The server analyzes this data and automatically generates a report containing the inspection results and any detected defects. This report becomes an important source of information for managers to view and take any necessary corrective measures promptly.

[0596] Examples of prompts using generative AI models include:

[0597] Generative AI prompts for work report generation systems:

[0598] 1. Please generate a work report based on the following text data.

[0599] 2. It should include a summary of the work, any issues found, and any suggested improvements.

[0600] 3. The output format is a Microsoft Word document.

[0601] In this way, the system of the present invention contributes to efficient management of factory work and improved quality.

[0602] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0603] Step 1:

[0604] Data collection by factory robots

[0605] input:

[0606] The user gives instructions to the factory robot to start work, and the robot collects video, audio, and image data using a camera, microphone, and image capture device.

[0607] output:

[0608] Data files for video (mp4 format), audio (wav format), and images (png or jpg format) are generated.

[0609] Specific behavior:

[0610] The robot's camera captures the work scene.

[0611] The microphone records your voice as you work.

[0612] An image capture device periodically takes still images.

[0613] Step 2:

[0614] Uploading data

[0615] input:

[0616] The video, audio, and image data collected by the robot is managed on a terminal.

[0617] output:

[0618] The data file is uploaded to the server.

[0619] Specific behavior:

[0620] The terminal receives data from the robot.

[0621] The device uploads the data to a server via the Internet.

[0622] Step 3:

[0623] Data analysis

[0624] input:

[0625] Video, audio, and image data uploaded to the server.

[0626] output:

[0627] Text data of character information and action information analyzed by the server.

[0628] Specific behavior:

[0629] The server analyzes the video data and extracts motion information for each specified frame.

[0630] The voice data is converted into text using a voice recognition engine (Google Cloud Speech-to-Text).

[0631] Text information is extracted from the image data using an optical character recognition engine (Tesseract OCR).

[0632] Step 4:

[0633] Material generation

[0634] input:

[0635] Text data of character information and action information analyzed by the server.

[0636] output:

[0637] Automatically generated work reports (Microsoft Word documents) based on predefined templates.

[0638] Specific behavior:

[0639] The server arranges the extracted text information according to a defined template.

[0640] Automatically generate work reports that include work details, issues detected, and suggested improvements.

[0641] Step 5:

[0642] Storage and distribution of materials

[0643] input:

[0644] Automatically generated work report (Microsoft Word document).

[0645] output:

[0646] Work reports stored in the server's cloud storage and distributed to stakeholders.

[0647] Specific behavior:

[0648] The server stores the generated work report in cloud storage.

[0649] Users can access the saved reports and distribute them to interested parties as needed.

[0650] Digital signature functionality is used to ensure the authenticity of reports.

[0651] The above steps create a system that automatically records factory work and creates reports, which improves work efficiency and strengthens quality control.

[0652] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0653] This invention combines a system that automatically generates materials based on user-provided video, audio, images, and text with an emotion engine that recognizes the user's emotions, enabling it to adjust the tone and content of the materials according to the user's emotional state. This system analyzes input data, extracts the user's emotional information, and automatically creates materials according to defined templates, significantly reducing the effort required to create materials and providing materials with more appropriate content.

[0654] System Overview

[0655] The system consists of the following major components:

[0656] 1. Data Entry Component

[0657] It provides an interface that allows users to input the content they want to document in the form of video, audio, images, and text.

[0658] Users log in to the system's portal and upload data.

[0659] 2. Data Analysis Component

[0660] The server analyzes the uploaded data, converting audio and video data into text using voice recognition technology, and extracting text information from image data using optical character recognition technology.

[0661] An emotion engine is used to recognize the user's emotions from video and audio, and organizes the emotional information together with text data.

[0662] 3. Document Generation Component

[0663] The server automatically generates materials based on the analyzed text data and emotional information according to predefined templates.

[0664] Adjust the tone and content of your materials depending on the user's emotional state.

[0665] 4. Material Storage and Distribution Component

[0666] The server stores the generated materials in the user's personal or project folder for easy access by the user.

[0667] Users can download the generated materials from within the system and share them with relevant parties as needed.

[0668] Specific processing flow

[0669] Data entry examples

[0670] For example, consider a case where a user wants to upload a recording of a meeting to the system. The user logs in to the system, selects the recording file, and clicks the upload button. The device then sends the recording file to the server and notifies the user when the upload is complete.

[0671] Specific examples of data analysis

[0672] Once the recording file is uploaded to the server, the server converts the voice data into text using speech recognition technology. At the same time, the emotion engine analyzes the tone, pitch, and speed of the voice to recognize the user's emotions. For example, if someone expresses strong interest or dissatisfaction during a meeting, that emotional information is recorded along with the text data.

[0673] Example of data generation

[0674] The analyzed text data and emotional information are organized and arranged according to a predefined meeting minutes template. The server automatically generates minutes based on the emotional information, emphasizing specific remarks and discussion points in an appropriate tone. For example, points that users showed strong interest in are recorded in particular detail.

[0675] Examples of material preservation and distribution

[0676] The automatically generated meeting minutes are saved in the user's personal folder by the server. Once saved, the user is notified and can download the minutes by logging back into the system. The downloaded minutes can then be distributed to relevant parties via email or the system's sharing function.

[0677] In this way, the present invention automates each process of document creation and further utilizes user emotional information to generate more accurate and appropriate documents, significantly improving work efficiency.

[0678] The processing flow will be explained below.

[0679] Step 1:

[0680] The user logs into the system's captive portal.

[0681] Step 2:

[0682] The user selects video, audio, image, or text data and clicks the upload button.

[0683] Step 3:

[0684] The terminal transmits the selected file to the server.

[0685] Step 4:

[0686] The server receives the uploaded data and identifies the data type (video, audio, image, text).

[0687] Step 5:

[0688] The server applies automatic speech recognition (ASR) technology to the audio and video files, converting the audio data into text.

[0689] Step 6:

[0690] The server applies optical character recognition (OCR) technology to the image file to extract text information from the image.

[0691] Step 7:

[0692] The server applies an emotion engine to the video or audio file to recognize the user's emotion. For example, the emotion engine analyzes the tone, pitch, and speed of the voice to identify whether the user is happy, angry, or surprised.

[0693] Step 8:

[0694] The server analyzes the text data and emotional information to extract sentence structure, keywords, and key points. The emotional information is used to emphasize and refine specific parts.

[0695] Step 9:

[0696] The server integrates all the analyzed text data and emotional information and organizes it into a single document.

[0697] Step 10:

[0698] The server automatically generates materials using the integrated text data and emotional information according to predefined templates, adjusting the tone and content of the materials by, for example, highlighting parts that express strong emotions.

[0699] Step 11:

[0700] The server stores the generated materials in the user's personal or project folder.

[0701] Step 12:

[0702] The server notifies the user when the document is complete.

[0703] Step 13:

[0704] The user logs back into the system to view and download the completed document.

[0705] Step 14:

[0706] Users can share the generated materials with stakeholders as needed by creating a sharing link within the system or sending it via email.

[0707] Example 2

[0708] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0709] Conventional document creation systems have been unable to generate documents that take the user's emotions into account, and as a result, the automatically generated documents often do not match the user's intentions or emotions. In addition, because many parts of the system rely on manual work, document creation takes time and effort.

[0710] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for a user to input video, audio, still image, and document data, means for analyzing the input data and extracting text information, means for recognizing the user's emotional information and adjusting the tone and content of the materials based on the emotional information, means for automatically generating materials using templates defined based on the extracted text information and emotional information, and means for saving and distributing the generated materials. This makes it possible to generate highly accurate materials that reflect the user's emotions.

[0711] "User" refers to any individual or organization that uses the System to input data and generate or download materials.

[0712] "Video" refers to digital or analog data that records moving images.

[0713] "Audio" refers to digital or analog data that records human speech or other sounds.

[0714] A "still image" refers to a single piece of image data that does not move.

[0715] "Document data" refers to data in the form of text files or documents that contain character information.

[0716] A "server" refers to a computer system that performs processes such as data analysis and automatic generation of materials.

[0717] "Template" refers to a defined format or layout used to automatically generate materials.

[0718] "Voice recognition technology" refers to the technology that analyzes voice data and converts it into text data.

[0719] "Optical character recognition technology (OCR)" refers to the technology of extracting character information from image data.

[0720] "Emotional information" refers to data obtained by analyzing the user's emotional state.

[0721] The "tone of a document" refers to the atmosphere and tone of the content and presentation of the document.

[0722] "Document generation" refers to the process of creating documents based on analyzed data.

[0723] "Preservation" refers to the keeping of generated materials in appropriate storage.

[0724] "Distribution" refers to providing generated materials to other users or interested parties.

[0725] This invention combines a system that automatically generates materials based on video, audio, still images, and text data provided by users with an emotion engine that recognizes the user's emotions, giving it the ability to adjust the tone and content of the materials according to the user's emotional state. This system analyzes the input data, extracts the user's emotional information, and automatically creates materials according to defined templates, significantly reducing the effort required to create materials and providing materials with more appropriate content.

[0726] System configuration

[0727] The system consists of the following major components:

[0728] 1. Data Entry Component

[0729] It provides an interface that allows users to input the content they want to document in the form of video, audio, still images, or documents.

[0730] Users log in to the system's portal and upload data.

[0731] 2. Data Analysis Component

[0732] The server analyzes the uploaded data, converting audio and video data into text using voice recognition technology, and extracting text information from still image data using optical character recognition (OCR).

[0733] An emotion engine is used to recognize the user's emotions from video and audio, and organizes the emotional information together with text data.

[0734] 3. Document Generation Component

[0735] The server automatically generates materials based on the analyzed text data and emotional information according to predefined templates.

[0736] Adjust the tone and content of your materials depending on the user's emotional state.

[0737] 4. Material Storage and Distribution Component

[0738] The server stores the generated materials in the user's personal or project folder for easy access by the user.

[0739] Users can download the generated materials from within the system and share them with relevant parties as needed.

[0740] Detailed component description

[0741] 1. Data Entry Component

[0742] Users first log in to the system's portal and upload the data they want to document (video, audio, still images, documents). The device then sends the selected files to the server and notifies the user when the upload is complete.

[0743] For example, if a user wants to upload a recording file of a meeting to the system, the user logs in, selects the recording file, and clicks the upload button. The device sends the recording file to the server and notifies the user when the upload is complete.

[0744] Prompt statement example (data entry):

[0745] A user logs in to the system, clicks the "Upload Data" button on the portal to upload the recording file of the meeting, selects the recording file and uploads it.

[0746] 2. Data Analysis Component

[0747] The server analyzes the uploaded data and uses speech recognition technology (e.g., speech recognition API) to convert audio and video data into text, and optical character recognition (OCR) technology (e.g., OCR software) to extract text information from still image data.

[0748] The server uses an emotion engine (e.g., emotion analysis API) to recognize the user's emotions from video and audio data, and organizes the emotional information together with the text data.

[0749] For example, after the server receives the uploaded recording file, it calls a speech recognition API to convert the voice data into text. At the same time, it uses an emotion analysis API to analyze the tone, pitch, speed, etc. of the voice to determine the user's emotion. For example, it generates a result such as "The user is angry" and stores it in a database along with the text data.

[0750] Example prompt (Data Analysis):

[0751] The server uses a speech recognition API to convert the voice data into text and an emotion analysis API to analyze the user's emotions.

[0752] 3. Document Generation Component

[0753] The server automatically generates materials based on the analyzed text data and emotion information according to a predefined template (e.g., document template).

[0754] The server generates materials based on the emotional information, emphasizing specific statements and discussion points in an appropriate tone. For example, parts that the user expressed strong interest in will be recorded in particular detail.

[0755] For example, the server reads the parsed text data and sentiment information, inserts these data into a template defined using the document template API, and the generated document is automatically saved as a "meeting minutes" file.

[0756] Example prompt (data generation):

[0757] The server organizes the analyzed text data and emotional information based on the document template API and automatically generates meeting minutes.

[0758] 4. Material Storage and Distribution Component

[0759] The server saves the generated materials in the user's personal folder or project folder (e.g., cloud storage service). Once the saving is complete, the user is notified.

[0760] Users can re-login to the system, access their personal folders, and download documents, and also share documents with relevant parties.

[0761] As a concrete example, the server uses a cloud storage API to save the generated "meeting minutes" file in the user's personal folder. After saving is complete, the server sends a notification to the user's device. The user then logs in again and downloads the document from their personal folder. The document can also be shared by clicking the "Share" button and entering the email addresses of those involved.

[0762] Example prompt (storage and distribution):

[0763] The server saves the automatically generated meeting minutes in cloud storage and notifies the user after saving is complete.

[0764] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0765] Step 1: Data entry

[0766] Specific explanation

[0767] Users log in to the system's portal and upload the data they want to document (video, audio, still images, documents). The terminal sends the selected files to the server and notifies the user when the upload is complete. The input is the file specified by the user, and the output is the data stored on the server.

[0768] Specific actions

[0769] After logging in, users can click the "Select File" button in the upload interface to select the file they want to document. By pressing the "Start Upload" button, the device will send the file to the server and display a progress bar. After the upload is complete, the device will display a notification message saying "Upload Completed."

[0770] Step 2: Data analysis

[0771] Specific explanation

[0772] The server analyzes the uploaded data. It uses voice recognition technology (e.g., voice recognition API) to convert audio and video data into text, and optical character recognition (OCR) technology (e.g., OCR software) to extract text information from still image data. It then uses an emotion engine (e.g., emotion analysis API) to recognize the user's emotions from the video and audio, and organizes the emotional information along with the text data. The input is the data stored on the server, and the output is the analyzed text data and emotional information.

[0773] Specific actions

[0774] After the server receives the uploaded audio file, it calls a speech recognition API to convert the audio data into text. At the same time, it uses an emotion analysis API to analyze the tone, pitch, speed, etc. of the voice to determine the user's emotion. For example, it generates a result such as "The user is angry" and stores it in the database along with the text data. For image files, it uses OCR software to extract characters and stores the analysis results in the database as text data.

[0775] Adding Inputs and Outputs

[0776] The input (data stored on the server) is audio files, image files, and document files, and the output (internal server data) is text data and emotional information obtained through analysis.

[0777] Step 3: Generate materials

[0778] Specific explanation

[0779] The server automatically generates materials based on the analyzed text data and emotional information according to a predefined template (e.g., a document template). The content of the materials is adjusted according to the user's emotional state. The input is the text data and emotional information stored on the server, and the output is the generated materials.

[0780] Specific actions

[0781] The server reads the parsed text data and emotion information and inserts this data into a template defined using the document template API. For example, it increases the font size or changes the color to highlight parts where the user expressed strong interest or anger. The generated document is saved as a "meeting minutes" file.

[0782] Adding Inputs and Outputs

[0783] The input (internal data of the server) is analyzed text data and emotion information, and the output (generated file) is material generated based on a template.

[0784] Step 4: Preservation and distribution

[0785] Specific explanation

[0786] The server saves the generated materials in the user's personal folder or project folder. Once saving is complete, a notification is sent to the user, who can then re-login to the system to download the materials. It also provides a function to share the materials with relevant parties. The input is the generated materials, and the output is the URL of the saved materials or a notification that saving has been completed.

[0787] Specific actions

[0788] The server uses the cloud storage API to save the generated "meeting minutes" file in the user's personal folder. After saving is complete, the server sends a notification to the user's device. The user can also log in to the system again, access their personal folder, and download the materials. The user can share the materials by clicking the "Share" button and entering the email addresses of the relevant parties.

[0789] Adding Inputs and Outputs

[0790] The input (the generated file) is the generated material, and the output (a user-accessible link or distributable data) is the URL of the material stored in cloud storage or a notification that the storage is complete.

[0791] (Application example 2)

[0792] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0793] Conventional document generation systems were capable of extracting textual information from user input data, but were unable to generate documents that reflected emotional information. As a result, users' intentions and nuances were sometimes not accurately conveyed, making it difficult to create more effective advertising proposals and documents. In particular, in marketing and advertising campaigns, proposals that capture the emotions of target users are required, so solving this problem is an urgent task.

[0794] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0795] In this invention, the server includes means for inputting video, audio, image, and text data, means for analyzing the input data and extracting character information and emotional information, means for automatically generating materials using templates defined based on the extracted character information and emotional information, and means for adjusting the tone and content of the materials based on the emotional information, and means for saving and distributing the generated materials. This enables the creation of more effective and detailed advertising proposal materials and materials that reflect the user's emotions.

[0796] "Motion picture" is a data format in which visual content is displayed continuously.

[0797] "Audio" is a data format that includes auditory information such as speech and sound effects.

[0798] An "image" is a data format for displaying fixed visual content.

[0799] "Text" is a data format that includes characters such as alphabets and kanji.

[0800] "Emotion information" is data that indicates the user's feelings and emotions, and is analyzed from the input data.

[0801] A "template" is a format used as a standard when generating materials.

[0802] "Materials" refers to documents or presentations created for a specific purpose or to organize information.

[0803] "Tone" refers to the overall atmosphere or mood of a document or piece of writing.

[0804] "Storage" refers to the act of recording generated materials in a database or file system.

[0805] "Distribution" refers to the act of sending or making available generated materials to other users or systems.

[0806] The present invention relates to a system for automatically generating proposal materials for an advertising campaign based on various input data. Hereinafter, an embodiment of the present invention will be described in detail.

[0807] First, the system provides an interface for inputting video, audio, image, and text data. Users log in to the system and upload this data, which is then sent to the server as the basis for creating materials. The server analyzes the input data and extracts text and emotional information. Audio data is converted into text using voice recognition technology, and image data is used to extract text information using optical character recognition technology.

[0808] At this time, emotional information is extracted from the analyzed audio and video using an emotion engine. This emotion engine analyzes the user's voice characteristics, such as tone, pitch, and speed, to recognize their emotional state. This emotional information and text information are then organized by the server.

[0809] The server then automatically generates ad campaign proposals using predefined templates based on the organized text and emotional information. The tone and content of the proposals are adjusted based on the user's emotional state. For example, specific ad copy or visuals may be emphasized based on areas of high interest.

[0810] The generated materials are stored on the server and made easily accessible to users, who can also download the materials from within the system and share them with interested parties.

[0811] The main hardware and software used in implementing the system are as follows:

[0812] Hardware: PC or smartphone with internet connection

[0813] Software: Python, cv2 library, speech_recognition library, transformers library

[0814] As a concrete example, consider a scenario in which a user wants to upload a recording of a meeting to a system. The user logs in to the system, selects the recording file, and clicks the upload button. The device then sends the recording file to the server. Once the upload is complete, the system converts the recording file into text using speech recognition technology, and automatically generates proposal materials for an advertising campaign based on the text data and emotional information.

[0815] An example prompt might look like this:

[0816] "Create an advertising campaign pitch from these voice memos. The memos include product features and highlights."

[0817] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0818] Step 1:

[0819] The user logs in to the system. The user enters authentication information on the provided login screen and accesses the system. This operation allows the user to specify which project to upload data to.

[0820] Step 2:

[0821] The user uploads video, audio, image, or text data. The user selects the data file on the upload screen and clicks the send button. The input data is sent from the device to the server. The input data format includes mp4, wav, jpg, pdf, txt, etc.

[0822] Step 3:

[0823] The server receives the uploaded data. The received data is first saved in a temporary location before being analyzed. For example, if audio data is provided in wav format, it will be saved in a specified directory.

[0824] Step 4:

[0825] The server analyzes the data and extracts text and emotional information. Specifically, it converts audio data into text using speech recognition technology, and extracts text from image data using optical character recognition (OCR). It then uses an emotion engine to analyze the tone, pitch, and speed of the audio and video to recognize the emotional state. In this step, audio or image data is received as input, and text and emotional information are generated as output.

[0826] Step 5:

[0827] The server automatically generates materials using templates defined based on text and emotional information. The server arranges the necessary data appropriately according to the template and adjusts the tone and content based on emotional information. For example, particularly detailed explanations are added to parts that show strong user interest. In this step, organized text and emotional information are received as input, and advertising campaign proposal materials are generated as output.

[0828] Step 6:

[0829] The server saves and distributes the generated materials. The automatically generated materials are saved in the user's personal folder or project folder. After saving is complete, a notification is sent to the user, who can then re-login to the system and download the materials. Furthermore, the user can share the downloaded materials with relevant parties via email or the sharing function. In this step, the generated materials are received as input, and saving and distribution are performed as output.

[0830] In this way, efficient and effective material creation is achieved through the specific processing steps of the system that automatically generates advertising campaign proposal materials that reflect user emotional information.

[0831] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0832] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0833] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0834] [Third embodiment]

[0835] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0836] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0837] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0838] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0839] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0840] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0841] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0842] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0843] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0844] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0845] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0846] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0847] The present invention is a system that automatically generates materials based on video, audio, images, and text provided by users. This system analyzes input data and automatically creates materials according to defined templates, significantly reducing the effort required for creating materials. Below, the program processing of this system is explained in natural language, and an embodiment is shown with specific examples.

[0848] System Overview

[0849] The system consists of the following major components:

[0850] 1. Data Entry Component

[0851] It provides an interface that allows users to input the content they want to document in the form of video, audio, images, and text.

[0852] Specifically, users log in to the system's portal and upload data.

[0853] 2. Data Analysis Component

[0854] The server analyzes the uploaded data, converting audio and video data into text using voice recognition technology, and extracting text information from image data using optical character recognition technology.

[0855] After analysis, the obtained text data is organized and integrated into a single text document.

[0856] 3. Document Generation Component

[0857] The server automatically generates materials based on the analyzed text data according to predefined templates.

[0858] For example, format it into meeting minutes, business trip reports, knowledge sharing materials, etc.

[0859] 4. Material Storage and Distribution Component

[0860] The server stores the generated materials in the user's personal or project folder for easy access by the user.

[0861] Users can download the generated materials from within the system and share them with relevant parties as needed.

[0862] Specific processing flow

[0863] Data entry examples

[0864] For example, suppose a user wants to upload a video of a presentation they gave on a business trip. After logging in to the system, the user selects the video file and clicks the upload button. At this time, the device sends the video file to the server and notifies the user when the upload is complete.

[0865] Specific examples of data analysis

[0866] When a video file is uploaded to the server, the server converts the audio in the video into text using speech recognition technology. If the audio data is in English, a speech recognition engine for the corresponding language is used to classify and convert the spoken content for each speaker. In addition, optical character recognition technology is used to extract text information from slide images that may be included in the video.

[0867] Example of data generation

[0868] The analyzed text data is organized and arranged according to a predefined business trip report template. The server automatically generates a report that highlights the presentation's main points and conclusions. If necessary, it also adds metadata such as the presentation date and trip destination.

[0869] Examples of material preservation and distribution

[0870] The server saves the automatically generated trip report in the user's personal folder. Once the save is complete, the user is notified and can download the report by logging back into the system. The downloaded report can be distributed to relevant parties via email or the system's sharing function.

[0871] In this way, the present invention automates each process of creating materials, thereby reducing the burden on users and significantly improving work efficiency.

[0872] The processing flow will be explained below.

[0873] Step 1:

[0874] The user logs into the system's captive portal.

[0875] Step 2:

[0876] The user selects video, audio, image, or text data and clicks the upload button.

[0877] Step 3:

[0878] The terminal transmits the selected file to the server.

[0879] Step 4:

[0880] The server receives the uploaded data and identifies the data type (video, audio, image, text).

[0881] Step 5:

[0882] The server applies automatic speech recognition (ASR) technology to the audio and video files, converting the audio data into text.

[0883] Step 6:

[0884] The server applies optical character recognition (OCR) technology to the image file to extract text information from the image.

[0885] Step 7:

[0886] The server analyzes the text data and extracts sentence structure, keywords, and important points.

[0887] Step 8:

[0888] The server consolidates all the analyzed text data and organizes it into a single document.

[0889] Step 9:

[0890] The server automatically generates materials using the integrated text data according to a predefined template.

[0891] Step 10:

[0892] The server stores the generated materials in the user's personal or project folder.

[0893] Step 11:

[0894] The server notifies the user when the document is complete.

[0895] Step 12:

[0896] The user logs back into the system to view and download the completed document.

[0897] Step 13:

[0898] The user shares the generated materials with the relevant parties as needed.

[0899] Example 1

[0900] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0901] In today's business environment, there is a need to organize large amounts of data and create documents quickly and accurately. However, when dealing with data in various formats (video, audio, images, text), much of the processing is done manually, which requires time and effort, resulting in a decrease in work efficiency. Furthermore, errors are likely to occur during the data analysis and document creation process, resulting in inconsistent quality of the results. A system that can solve these problems and improve work efficiency and document quality is needed.

[0902] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0903] In this invention, the server includes: a means for a user to input data in the form of video, audio, image, or text; a means for the server to analyze the input data and extract character information using voice recognition technology and optical character recognition technology; a means for the server to automatically generate materials based on the extracted character information using a defined template; and a means for the server to store the generated materials and notify and distribute them to users. This makes it possible to effectively analyze and integrate various data formats, quickly and automatically generate high-quality materials, and significantly reduce the user's workload.

[0904] "User" means any person or entity that wishes to use the System to input data and generate materials.

[0905] "Server" refers to a computer system that receives data from users, analyzes it, and automatically generates materials.

[0906] "Data" means information, whether in the form of video, audio, images, or text, entered into the system by a user.

[0907] "Means for input" refers to the interface and functionality for users to upload data into the system.

[0908] "Analyzing" refers to the process in which the server uses voice recognition technology, optical character recognition technology, and the like to extract text information from the input data.

[0909] "Voice recognition technology" refers to the technology that analyzes voice data and converts the content into text format.

[0910] "Optical character recognition technology" refers to the technology that analyzes character information contained in image data and converts it into text format.

[0911] "Defined template" refers to a pre-set format or style used when automatically generating materials.

[0912] The "means for automatically generating materials" refers to a process in which the server creates materials using a template defined based on the extracted character information.

[0913] "Storing" refers to the server storing the generated material in its own storage device in order to provide it to the user.

[0914] "Distributing" refers to the server notifying the user of the generated material and making it accessible to the user.

[0915] This invention is a system that automatically generates materials based on video, audio, images, and text provided by users. The system consists of the following main components:

[0916] Data Entry Components

[0917] Users log in to the system's portal and use an interface to input the content they want to document in the form of video, audio, images, and text. Specifically, users select a file and click the upload button to send the data from their device to the server.

[0918] Data Analysis Components

[0919] The server analyzes the uploaded data. For example, for video and audio data, speech recognition technology (e.g., Google Speech-to-Text API) is used to convert the speech into text. For image data, optical character recognition technology (e.g., Google Cloud Vision API) is used to extract text information. The resulting text data is organized and integrated into a single text document.

[0920] Document generation component

[0921] The server automatically generates documents based on the analyzed text data according to predefined templates (e.g., Microsoft Word templates), for example, meeting minutes, business trip reports, knowledge sharing materials, etc.

[0922] Material storage and distribution component

[0923] The server saves the generated materials in the user's personal or project folder, and once the saving is complete, the server notifies the user and provides a download link for the materials, allowing the user to easily access the materials.

[0924] Specific processing flow example

[0925] 1. Example of data entry

[0926] If a user wants to upload a video of a presentation they gave on a business trip, they log in to the system, select "presentation.mp4," and click the upload button. The device sends this video file to the server, and the server sends a notification that it has been received. The user sees a message saying, "Upload completed."

[0927] 2. Specific examples of data analysis

[0928] The server analyzes "presentation.mp4" and converts the audio into text using the Google Speech-to-Text API. As a result of the analysis, text data such as "The main points of today's presentation are..." is extracted. Furthermore, text information such as "Figure 1: Sales Trends" is extracted from the slide images in the video using the Google Cloud Vision API, and all text data is integrated into a single document.

[0929] 3. Specific examples of data generation

[0930] The server generates a business trip report template based on the integrated text data. The content includes sections such as "Presentation Overview," "Main Points," and "Conclusion." Metadata such as "Date: October 10, 2023" and "Location: Tokyo" are added.

[0931] 4. Examples of material preservation and distribution

[0932] The generated business trip report is saved in the user's personal folder as "report20231010.docx." The server sends the user a notification that "The document has been generated and is available for download." The user then logs in to the system again, clicks the download link, and downloads the report. The downloaded report can be shared with team members via email.

[0933] Prompt Sentence Examples

[0934] For example, the following prompts can explain the process of uploading a video of a presentation a user gave on a business trip, using speech recognition technology to convert the audio in the video into text, and automatically generating materials based on a template for a business trip report:

[0935] Simply upload a video of your business trip presentation to the system. We use speech recognition technology to convert the audio in the video into text, and automatically generate a business trip report template. The report will highlight key points and conclusions, and include metadata about the presentation date and the business trip destination.

[0936] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0937] Step 1: Data entry

[0938] Input: Users log in to the system's portal and select the video, audio, image, or text data they wish to document.

[0939] Specific operation: The user selects a file on the upload screen and clicks the upload button, at which point the device sends the selected file to the server.

[0940] Output: The server receives the file and sends a notification to the user that the upload is complete. The user sees the message "Upload completed."

[0941] Step 2: Receiving and storing data

[0942] Input: Files sent from the device (video, audio, image, text data).

[0943] Specific operation: The server stores the received file in temporary storage, which is used for subsequent analysis.

[0944] Output: The path and filename of the saved file. For example, / tmp / presentation.mp4.

[0945] Step 3: Data analysis (speech recognition)

[0946] Input: A video or audio file stored in temporary storage.

[0947] Specific operation: The server converts the voice data into text using speech recognition technology (e.g., Google Speech-to-Text API).

[0948] Data processing: Convert audio data in video or audio files into text format.

[0949] Output: The converted text data, e.g., "The main points of the presentation are...".

[0950] Step 4: Data Analysis (Optical Character Recognition)

[0951] Input: An image file or a frame from a video file saved in temporary storage.

[0952] Specific operation: The server uses optical character recognition technology (e.g., Google Cloud Vision API) to extract text information from the image.

[0953] Data processing: Converts text information in image files into text format.

[0954] Output: Extracted text data. For example, "Figure 1: Sales Trends."

[0955] Step 5: Data Integration

[0956] Input: Text data extracted using speech recognition and optical character recognition techniques.

[0957] Specific operation: The server organizes and integrates the multiple text data obtained into a single text document.

[0958] Data processing: Integrate scattered text data and organize it into documents.

[0959] Output: A consolidated text document, e.g., "Presentation Report Text Data".

[0960] Step 6: Generate materials

[0961] Input: A consolidated text document.

[0962] Specific operation: The server uses a predefined template to automatically generate documents based on the integrated text data, for example, using a Microsoft Word template.

[0963] Data processing: Place text documents according to templates and add required metadata (e.g. date, location, etc.).

[0964] Output: An automatically generated document file, for example, a business trip report (report20231010.docx).

[0965] Step 7: Preservation of materials and user notification

[0966] Input: Auto-generated documentation file.

[0967] Specific operation: The server saves the generated materials in the user's personal folder or project folder, and notifies the user when the saving is complete.

[0968] Output: The path and download link of the saved document, for example, / user / reports / report20231010.docx.

[0969] Step 8: Download and share materials

[0970] Input: The download link provided to the user.

[0971] Specific operation: The user logs in to the system again, clicks the download link to download the document, and then shares the document with relevant parties as needed.

[0972] Output: Downloaded documents and share them. For example, send a business trip report to a team member via email.

[0973] By following the above steps, this system automates the process from analyzing input data to generating, saving, and distributing materials, significantly reducing the user's workload.

[0974] (Application example 1)

[0975] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0976] Recording work processes and creating reports in factories requires a huge amount of time and effort when done manually, and there is a high risk of errors and information omissions. It also presents challenges in accurately understanding work content and quickly troubleshooting. Under these circumstances, automating work records and quickly generating reports is required to improve work efficiency and quality control throughout the factory.

[0977] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0978] In this invention, the server includes means for inputting video, audio, image, and text data, means for analyzing the input data and extracting character information, means for automatically generating materials using templates defined based on the extracted character information, means for saving and distributing the generated materials, and means for analyzing data captured by a factory work robot and automatically generating work reports. This automates the recording of work processes and the creation of reports in a factory, making it possible to improve work efficiency and strengthen quality control.

[0979] "Video" is data that records moving images or videos.

[0980] "Audio" is data that is a recording of a human voice or other sounds.

[0981] An "image" is data that records still visual information.

[0982] "Text" is information expressed in characters.

[0983] "Input means" refers to a method or device that allows a user or robot to provide video, audio, image, or text data to the system.

[0984] "Means of analysis" are techniques and methods for extracting useful information from the provided data.

[0985] "Text information" is information expressed as text.

[0986] "Extraction means" refers to the technology or equipment used to extract the necessary information from the data.

[0987] A "defined template" is a set of rules or formats for creating materials according to a specific format or structure.

[0988] "Means for automatically generating materials" refers to techniques or methods for automatically creating materials based on extracted information in accordance with predefined templates.

[0989] "Means for storing and distributing" refers to the techniques and methods for storing the automatically generated materials and distributing them to users who need them.

[0990] A "factory work robot" is a robotic device that automatically performs work in a factory.

[0991] A "work report" is a document that lists details of the work, results, problems, etc.

[0992] The following describes in detail an embodiment of the present invention. The present invention is a system that analyzes video, audio, and image data captured by a factory work robot and automatically generates a work report based on the data. The system of the present invention includes means for inputting video, audio, image, and text data, and means for analyzing each data and extracting text information. It also includes means for automatically generating materials using templates defined based on the extracted text information, and means for saving and distributing the generated materials. The specific configuration and operation of each means are described below.

[0993] First, the video, audio, and image data captured by the factory robot during work is sent to the system's server via input means, which include the robot's camera, microphone, and image capture device. The user then gives the robot appropriate instructions to collect data.

[0994] The server then analyzes the video, audio, and image data and extracts the necessary text information. This analysis uses the following techniques:

[0995] For voice recognition, a voice recognition engine (e.g., "Google Cloud Speech-to-Text") is used.

[0996] To extract text information from image data, an optical character recognition engine (e.g., "Tesseract OCR") is used.

[0997] A video analysis engine (e.g., OpenCV and deep learning models) is used to analyze the movements.

[0998] After data analysis, the extracted text information is automatically organized and consolidated by the server based on a predefined work report template, which includes, for example, the steps performed by the robot, any problems detected, and any suggested improvements.

[0999] Finally, the generated work report is stored by the server and distributed to the relevant parties as needed. The authenticity of the stored report may be verified by a digital signature, ensuring the reliability of the report content. Through this storage and distribution method, users and administrators can easily access and share the report.

[1000] A concrete example of this would be the following scenario. Suppose a robot is inspecting products on a factory production line. The robot captures video and audio during the inspection and sends the data to a server. The server analyzes this data and automatically generates a report containing the inspection results and any detected defects. This report becomes an important source of information for managers to view and take any necessary corrective measures promptly.

[1001] Examples of prompts using generative AI models include:

[1002] Generative AI prompts for work report generation systems:

[1003] 1. Please generate a work report based on the following text data.

[1004] 2. It should include a summary of the work, any issues found, and any suggested improvements.

[1005] 3. The output format is a Microsoft Word document.

[1006] In this way, the system of the present invention contributes to efficient management of factory work and improved quality.

[1007] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1008] Step 1:

[1009] Data collection by factory robots

[1010] input:

[1011] The user gives instructions to the factory robot to start work, and the robot collects video, audio, and image data using a camera, microphone, and image capture device.

[1012] output:

[1013] Data files for video (mp4 format), audio (wav format), and images (png or jpg format) are generated.

[1014] Specific behavior:

[1015] The robot's camera captures the work scene.

[1016] The microphone records your voice as you work.

[1017] An image capture device periodically takes still images.

[1018] Step 2:

[1019] Uploading data

[1020] input:

[1021] The video, audio, and image data collected by the robot is managed on a terminal.

[1022] output:

[1023] The data file is uploaded to the server.

[1024] Specific behavior:

[1025] The terminal receives data from the robot.

[1026] The device uploads the data to a server via the Internet.

[1027] Step 3:

[1028] Data analysis

[1029] input:

[1030] Video, audio, and image data uploaded to the server.

[1031] output:

[1032] Text data of character information and action information analyzed by the server.

[1033] Specific behavior:

[1034] The server analyzes the video data and extracts motion information for each specified frame.

[1035] The voice data is converted into text using a voice recognition engine (Google Cloud Speech-to-Text).

[1036] Text information is extracted from the image data using an optical character recognition engine (Tesseract OCR).

[1037] Step 4:

[1038] Material generation

[1039] input:

[1040] Text data of character information and action information analyzed by the server.

[1041] output:

[1042] Automatically generated work reports (Microsoft Word documents) based on predefined templates.

[1043] Specific behavior:

[1044] The server arranges the extracted text information according to a defined template.

[1045] Automatically generate work reports that include work details, issues detected, and suggested improvements.

[1046] Step 5:

[1047] Storage and distribution of materials

[1048] input:

[1049] Automatically generated work report (Microsoft Word document).

[1050] output:

[1051] Work reports stored in the server's cloud storage and distributed to stakeholders.

[1052] Specific behavior:

[1053] The server stores the generated work report in cloud storage.

[1054] Users can access the saved reports and distribute them to interested parties as needed.

[1055] Digital signature functionality is used to ensure the authenticity of reports.

[1056] The above steps create a system that automatically records factory work and creates reports, which improves work efficiency and strengthens quality control.

[1057] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1058] This invention combines a system that automatically generates materials based on user-provided video, audio, images, and text with an emotion engine that recognizes the user's emotions, enabling it to adjust the tone and content of the materials according to the user's emotional state. This system analyzes input data, extracts the user's emotional information, and automatically creates materials according to defined templates, significantly reducing the effort required to create materials and providing materials with more appropriate content.

[1059] System Overview

[1060] The system consists of the following major components:

[1061] 1. Data Entry Component

[1062] It provides an interface that allows users to input the content they want to document in the form of video, audio, images, and text.

[1063] Users log in to the system's portal and upload data.

[1064] 2. Data Analysis Component

[1065] The server analyzes the uploaded data, converting audio and video data into text using voice recognition technology, and extracting text information from image data using optical character recognition technology.

[1066] An emotion engine is used to recognize the user's emotions from video and audio, and organizes the emotional information together with text data.

[1067] 3. Document Generation Component

[1068] The server automatically generates materials based on the analyzed text data and emotional information according to predefined templates.

[1069] Adjust the tone and content of your materials depending on the user's emotional state.

[1070] 4. Material Storage and Distribution Component

[1071] The server stores the generated materials in the user's personal or project folder for easy access by the user.

[1072] Users can download the generated materials from within the system and share them with relevant parties as needed.

[1073] Specific processing flow

[1074] Data entry examples

[1075] For example, consider a case where a user wants to upload a recording of a meeting to the system. The user logs in to the system, selects the recording file, and clicks the upload button. The device then sends the recording file to the server and notifies the user when the upload is complete.

[1076] Specific examples of data analysis

[1077] Once the recording file is uploaded to the server, the server converts the voice data into text using speech recognition technology. At the same time, the emotion engine analyzes the tone, pitch, and speed of the voice to recognize the user's emotions. For example, if someone expresses strong interest or dissatisfaction during a meeting, that emotional information is recorded along with the text data.

[1078] Example of data generation

[1079] The analyzed text data and emotional information are organized and arranged according to a predefined meeting minutes template. The server automatically generates minutes based on the emotional information, emphasizing specific remarks and discussion points in an appropriate tone. For example, points that users showed strong interest in are recorded in particular detail.

[1080] Examples of material preservation and distribution

[1081] The automatically generated meeting minutes are saved in the user's personal folder by the server. Once saved, the user is notified and can download the minutes by logging back into the system. The downloaded minutes can then be distributed to relevant parties via email or the system's sharing function.

[1082] In this way, the present invention automates each process of document creation and further utilizes user emotional information to generate more accurate and appropriate documents, significantly improving work efficiency.

[1083] The processing flow will be explained below.

[1084] Step 1:

[1085] The user logs into the system's captive portal.

[1086] Step 2:

[1087] The user selects video, audio, image, or text data and clicks the upload button.

[1088] Step 3:

[1089] The terminal transmits the selected file to the server.

[1090] Step 4:

[1091] The server receives the uploaded data and identifies the data type (video, audio, image, text).

[1092] Step 5:

[1093] The server applies automatic speech recognition (ASR) technology to the audio and video files, converting the audio data into text.

[1094] Step 6:

[1095] The server applies optical character recognition (OCR) technology to the image file to extract text information from the image.

[1096] Step 7:

[1097] The server applies an emotion engine to the video or audio file to recognize the user's emotion. For example, the emotion engine analyzes the tone, pitch, and speed of the voice to identify whether the user is happy, angry, or surprised.

[1098] Step 8:

[1099] The server analyzes the text data and emotional information to extract sentence structure, keywords, and key points. The emotional information is used to emphasize and refine specific parts.

[1100] Step 9:

[1101] The server integrates all the analyzed text data and emotional information and organizes it into a single document.

[1102] Step 10:

[1103] The server automatically generates materials using the integrated text data and emotional information according to predefined templates, adjusting the tone and content of the materials by, for example, highlighting parts that express strong emotions.

[1104] Step 11:

[1105] The server stores the generated materials in the user's personal or project folder.

[1106] Step 12:

[1107] The server notifies the user when the document is complete.

[1108] Step 13:

[1109] The user logs back into the system to view and download the completed document.

[1110] Step 14:

[1111] Users can share the generated materials with stakeholders as needed by creating a sharing link within the system or sending it via email.

[1112] Example 2

[1113] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1114] Conventional document creation systems have been unable to generate documents that take the user's emotions into account, and as a result, the automatically generated documents often do not match the user's intentions or emotions. In addition, because many parts of the system rely on manual work, document creation takes time and effort.

[1115] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for a user to input video, audio, still image, and document data, means for analyzing the input data and extracting text information, means for recognizing the user's emotional information and adjusting the tone and content of the materials based on the emotional information, means for automatically generating materials using templates defined based on the extracted text information and emotional information, and means for saving and distributing the generated materials. This makes it possible to generate highly accurate materials that reflect the user's emotions.

[1116] "User" refers to any individual or organization that uses the System to input data and generate or download materials.

[1117] "Video" refers to digital or analog data that records moving images.

[1118] "Audio" refers to digital or analog data that records human speech or other sounds.

[1119] A "still image" refers to a single piece of image data that does not move.

[1120] "Document data" refers to data in the form of text files or documents that contain character information.

[1121] A "server" refers to a computer system that performs processes such as data analysis and automatic generation of materials.

[1122] "Template" refers to a defined format or layout used to automatically generate materials.

[1123] "Voice recognition technology" refers to the technology that analyzes voice data and converts it into text data.

[1124] "Optical character recognition technology (OCR)" refers to the technology of extracting character information from image data.

[1125] "Emotional information" refers to data obtained by analyzing the user's emotional state.

[1126] The "tone of a document" refers to the atmosphere and tone of the content and presentation of the document.

[1127] "Document generation" refers to the process of creating documents based on analyzed data.

[1128] "Preservation" refers to the keeping of generated materials in appropriate storage.

[1129] "Distribution" refers to providing generated materials to other users or interested parties.

[1130] This invention combines a system that automatically generates materials based on video, audio, still images, and text data provided by users with an emotion engine that recognizes the user's emotions, giving it the ability to adjust the tone and content of the materials according to the user's emotional state. This system analyzes the input data, extracts the user's emotional information, and automatically creates materials according to defined templates, significantly reducing the effort required to create materials and providing materials with more appropriate content.

[1131] System configuration

[1132] The system consists of the following major components:

[1133] 1. Data Entry Component

[1134] It provides an interface that allows users to input the content they want to document in the form of video, audio, still images, or documents.

[1135] Users log in to the system's portal and upload data.

[1136] 2. Data Analysis Component

[1137] The server analyzes the uploaded data, converting audio and video data into text using voice recognition technology, and extracting text information from still image data using optical character recognition (OCR).

[1138] An emotion engine is used to recognize the user's emotions from video and audio, and organizes the emotional information together with text data.

[1139] 3. Document Generation Component

[1140] The server automatically generates materials based on the analyzed text data and emotional information according to predefined templates.

[1141] Adjust the tone and content of your materials depending on the user's emotional state.

[1142] 4. Material Storage and Distribution Component

[1143] The server stores the generated materials in the user's personal or project folder for easy access by the user.

[1144] Users can download the generated materials from within the system and share them with relevant parties as needed.

[1145] Detailed component description

[1146] 1. Data Entry Component

[1147] Users first log in to the system's portal and upload the data they want to document (video, audio, still images, documents). The device then sends the selected files to the server and notifies the user when the upload is complete.

[1148] For example, if a user wants to upload a recording file of a meeting to the system, the user logs in, selects the recording file, and clicks the upload button. The device sends the recording file to the server and notifies the user when the upload is complete.

[1149] Prompt statement example (data entry):

[1150] A user logs in to the system, clicks the "Upload Data" button on the portal to upload the recording file of the meeting, selects the recording file and uploads it.

[1151] 2. Data Analysis Component

[1152] The server analyzes the uploaded data and uses speech recognition technology (e.g., speech recognition API) to convert audio and video data into text, and optical character recognition (OCR) technology (e.g., OCR software) to extract text information from still image data.

[1153] The server uses an emotion engine (e.g., emotion analysis API) to recognize the user's emotions from video and audio data, and organizes the emotional information together with the text data.

[1154] For example, after the server receives the uploaded recording file, it calls a speech recognition API to convert the voice data into text. At the same time, it uses an emotion analysis API to analyze the tone, pitch, speed, etc. of the voice to determine the user's emotion. For example, it generates a result such as "The user is angry" and stores it in a database along with the text data.

[1155] Example prompt (Data Analysis):

[1156] The server uses a speech recognition API to convert the voice data into text and an emotion analysis API to analyze the user's emotions.

[1157] 3. Document Generation Component

[1158] The server automatically generates materials based on the analyzed text data and emotion information according to a predefined template (e.g., document template).

[1159] The server generates materials based on the emotional information, emphasizing specific statements and discussion points in an appropriate tone. For example, parts that the user expressed strong interest in will be recorded in particular detail.

[1160] For example, the server reads the parsed text data and sentiment information, inserts these data into a template defined using the document template API, and the generated document is automatically saved as a "meeting minutes" file.

[1161] Example prompt (data generation):

[1162] The server organizes the analyzed text data and emotional information based on the document template API and automatically generates meeting minutes.

[1163] 4. Material Storage and Distribution Component

[1164] The server saves the generated materials in the user's personal folder or project folder (e.g., cloud storage service). Once the saving is complete, the user is notified.

[1165] Users can re-login to the system, access their personal folders, and download documents, and also share documents with relevant parties.

[1166] As a concrete example, the server uses a cloud storage API to save the generated "meeting minutes" file in the user's personal folder. After saving is complete, the server sends a notification to the user's device. The user then logs in again and downloads the document from their personal folder. The document can also be shared by clicking the "Share" button and entering the email addresses of those involved.

[1167] Example prompt (storage and distribution):

[1168] The server saves the automatically generated meeting minutes in cloud storage and notifies the user after saving is complete.

[1169] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1170] Step 1: Data entry

[1171] Specific explanation

[1172] Users log in to the system's portal and upload the data they want to document (video, audio, still images, documents). The terminal sends the selected files to the server and notifies the user when the upload is complete. The input is the file specified by the user, and the output is the data stored on the server.

[1173] Specific actions

[1174] After logging in, users can click the "Select File" button in the upload interface to select the file they want to document. By pressing the "Start Upload" button, the device will send the file to the server and display a progress bar. After the upload is complete, the device will display a notification message saying "Upload Completed."

[1175] Step 2: Data analysis

[1176] Specific explanation

[1177] The server analyzes the uploaded data. It uses voice recognition technology (e.g., voice recognition API) to convert audio and video data into text, and optical character recognition (OCR) technology (e.g., OCR software) to extract text information from still image data. It then uses an emotion engine (e.g., emotion analysis API) to recognize the user's emotions from the video and audio, and organizes the emotional information along with the text data. The input is the data stored on the server, and the output is the analyzed text data and emotional information.

[1178] Specific actions

[1179] After the server receives the uploaded audio file, it calls a speech recognition API to convert the audio data into text. At the same time, it uses an emotion analysis API to analyze the tone, pitch, speed, etc. of the voice to determine the user's emotion. For example, it generates a result such as "The user is angry" and stores it in the database along with the text data. For image files, it uses OCR software to extract characters and stores the analysis results in the database as text data.

[1180] Adding Inputs and Outputs

[1181] The input (data stored on the server) is audio files, image files, and document files, and the output (internal server data) is text data and emotional information obtained through analysis.

[1182] Step 3: Generate materials

[1183] Specific explanation

[1184] The server automatically generates materials based on the analyzed text data and emotional information according to a predefined template (e.g., a document template). The content of the materials is adjusted according to the user's emotional state. The input is the text data and emotional information stored on the server, and the output is the generated materials.

[1185] Specific actions

[1186] The server reads the parsed text data and emotion information and inserts this data into a template defined using the document template API. For example, it increases the font size or changes the color to highlight parts where the user expressed strong interest or anger. The generated document is saved as a "meeting minutes" file.

[1187] Adding Inputs and Outputs

[1188] The input (internal data of the server) is analyzed text data and emotion information, and the output (generated file) is material generated based on a template.

[1189] Step 4: Preservation and distribution

[1190] Specific explanation

[1191] The server saves the generated materials in the user's personal folder or project folder. Once saving is complete, a notification is sent to the user, who can then re-login to the system to download the materials. It also provides a function to share the materials with relevant parties. The input is the generated materials, and the output is the URL of the saved materials or a notification that saving has been completed.

[1192] Specific actions

[1193] The server uses the cloud storage API to save the generated "meeting minutes" file in the user's personal folder. After saving is complete, the server sends a notification to the user's device. The user can also log in to the system again, access their personal folder, and download the materials. The user can share the materials by clicking the "Share" button and entering the email addresses of the relevant parties.

[1194] Adding Inputs and Outputs

[1195] The input (the generated file) is the generated material, and the output (a user-accessible link or distributable data) is the URL of the material stored in cloud storage or a notification that the storage is complete.

[1196] (Application example 2)

[1197] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1198] Conventional document generation systems were capable of extracting textual information from user input data, but were unable to generate documents that reflected emotional information. As a result, users' intentions and nuances were sometimes not accurately conveyed, making it difficult to create more effective advertising proposals and documents. In particular, in marketing and advertising campaigns, proposals that capture the emotions of target users are required, so solving this problem is an urgent task.

[1199] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1200] In this invention, the server includes means for inputting video, audio, image, and text data, means for analyzing the input data and extracting character information and emotional information, means for automatically generating materials using templates defined based on the extracted character information and emotional information, and means for adjusting the tone and content of the materials based on the emotional information, and means for saving and distributing the generated materials. This enables the creation of more effective and detailed advertising proposal materials and materials that reflect the user's emotions.

[1201] "Motion picture" is a data format in which visual content is displayed continuously.

[1202] "Audio" is a data format that includes auditory information such as speech and sound effects.

[1203] An "image" is a data format for displaying fixed visual content.

[1204] "Text" is a data format that includes characters such as alphabets and kanji.

[1205] "Emotion information" is data that indicates the user's feelings and emotions, and is analyzed from the input data.

[1206] A "template" is a format used as a standard when generating materials.

[1207] "Materials" refers to documents or presentations created for a specific purpose or to organize information.

[1208] "Tone" refers to the overall atmosphere or mood of a document or piece of writing.

[1209] "Storage" refers to the act of recording generated materials in a database or file system.

[1210] "Distribution" refers to the act of sending or making available generated materials to other users or systems.

[1211] The present invention relates to a system for automatically generating proposal materials for an advertising campaign based on various input data. Hereinafter, an embodiment of the present invention will be described in detail.

[1212] First, the system provides an interface for inputting video, audio, image, and text data. Users log in to the system and upload this data, which is then sent to the server as the basis for creating materials. The server analyzes the input data and extracts text and emotional information. Audio data is converted into text using voice recognition technology, and image data is used to extract text information using optical character recognition technology.

[1213] At this time, emotional information is extracted from the analyzed audio and video using an emotion engine. This emotion engine analyzes the user's voice characteristics, such as tone, pitch, and speed, to recognize their emotional state. This emotional information and text information are then organized by the server.

[1214] The server then automatically generates ad campaign proposals using predefined templates based on the organized text and emotional information. The tone and content of the proposals are adjusted based on the user's emotional state. For example, specific ad copy or visuals may be emphasized based on areas of high interest.

[1215] The generated materials are stored on the server and made easily accessible to users, who can also download the materials from within the system and share them with interested parties.

[1216] The main hardware and software used in implementing the system are as follows:

[1217] Hardware: PC or smartphone with internet connection

[1218] Software: Python, cv2 library, speech_recognition library, transformers library

[1219] As a concrete example, consider a scenario in which a user wants to upload a recording of a meeting to a system. The user logs in to the system, selects the recording file, and clicks the upload button. The device then sends the recording file to the server. Once the upload is complete, the system converts the recording file into text using speech recognition technology, and automatically generates proposal materials for an advertising campaign based on the text data and emotional information.

[1220] An example prompt might look like this:

[1221] "Create an advertising campaign pitch from these voice memos. The memos include product features and highlights."

[1222] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1223] Step 1:

[1224] The user logs in to the system. The user enters authentication information on the provided login screen and accesses the system. This operation allows the user to specify which project to upload data to.

[1225] Step 2:

[1226] The user uploads video, audio, image, or text data. The user selects the data file on the upload screen and clicks the send button. The input data is sent from the device to the server. The input data format includes mp4, wav, jpg, pdf, txt, etc.

[1227] Step 3:

[1228] The server receives the uploaded data. The received data is first saved in a temporary location before being analyzed. For example, if audio data is provided in wav format, it will be saved in the specified directory.

[1229] Step 4:

[1230] The server analyzes the data and extracts text and emotional information. Specifically, it converts audio data into text using speech recognition technology, and extracts text from image data using optical character recognition (OCR). It then uses an emotion engine to analyze the tone, pitch, and speed of the audio and video to recognize the emotional state. In this step, audio or image data is received as input, and text and emotional information are generated as output.

[1231] Step 5:

[1232] The server automatically generates materials using templates defined based on text and emotional information. The server arranges the necessary data appropriately according to the template and adjusts the tone and content based on emotional information. For example, particularly detailed explanations are added to parts that show strong user interest. In this step, organized text and emotional information are received as input, and advertising campaign proposal materials are generated as output.

[1233] Step 6:

[1234] The server saves and distributes the generated materials. The automatically generated materials are saved in the user's personal folder or project folder. After saving is complete, a notification is sent to the user, who can then re-login to the system and download the materials. Furthermore, the user can share the downloaded materials with relevant parties via email or the sharing function. In this step, the generated materials are received as input, and saving and distribution are performed as output.

[1235] In this way, efficient and effective material creation is achieved through the specific processing steps of the system that automatically generates advertising campaign proposal materials that reflect user emotional information.

[1236] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1237] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1238] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1239] [Fourth embodiment]

[1240] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1241] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1242] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1243] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1244] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1245] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1246] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1247] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1248] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1249] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1250] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1251] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1252] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1253] The present invention is a system that automatically generates materials based on video, audio, images, and text provided by users. This system analyzes input data and automatically creates materials according to defined templates, significantly reducing the effort required for creating materials. Below, the program processing of this system is explained in natural language, and an embodiment is shown with specific examples.

[1254] System Overview

[1255] The system consists of the following major components:

[1256] 1. Data Entry Component

[1257] It provides an interface that allows users to input the content they want to document in the form of video, audio, images, and text.

[1258] Specifically, users log in to the system's portal and upload data.

[1259] 2. Data Analysis Component

[1260] The server analyzes the uploaded data, converting audio and video data into text using voice recognition technology, and extracting text information from image data using optical character recognition technology.

[1261] After analysis, the obtained text data is organized and integrated into a single text document.

[1262] 3. Document Generation Component

[1263] The server automatically generates materials based on the analyzed text data according to predefined templates.

[1264] For example, format it into meeting minutes, business trip reports, knowledge sharing materials, etc.

[1265] 4. Material Storage and Distribution Component

[1266] The server stores the generated materials in the user's personal or project folder for easy access by the user.

[1267] Users can download the generated materials from within the system and share them with relevant parties as needed.

[1268] Specific processing flow

[1269] Data entry examples

[1270] For example, suppose a user wants to upload a video of a presentation they gave on a business trip. After logging in to the system, the user selects the video file and clicks the upload button. At this time, the device sends the video file to the server and notifies the user when the upload is complete.

[1271] Specific examples of data analysis

[1272] When a video file is uploaded to the server, the server converts the audio in the video into text using speech recognition technology. If the audio data is in English, a speech recognition engine for the corresponding language is used to classify and convert the spoken content for each speaker. In addition, optical character recognition technology is used to extract text information from slide images that may be included in the video.

[1273] Example of data generation

[1274] The analyzed text data is organized and arranged according to a predefined business trip report template. The server automatically generates a report that highlights the presentation's main points and conclusions. If necessary, it also adds metadata such as the presentation date and trip destination.

[1275] Examples of material preservation and distribution

[1276] The server saves the automatically generated trip report in the user's personal folder. Once the save is complete, the user is notified and can download the report by logging back into the system. The downloaded report can be distributed to relevant parties via email or the system's sharing function.

[1277] In this way, the present invention automates each process of creating materials, thereby reducing the burden on users and significantly improving work efficiency.

[1278] The processing flow will be explained below.

[1279] Step 1:

[1280] The user logs into the system's captive portal.

[1281] Step 2:

[1282] The user selects video, audio, image, or text data and clicks the upload button.

[1283] Step 3:

[1284] The terminal transmits the selected file to the server.

[1285] Step 4:

[1286] The server receives the uploaded data and identifies the data type (video, audio, image, text).

[1287] Step 5:

[1288] The server applies automatic speech recognition (ASR) technology to the audio and video files, converting the audio data into text.

[1289] Step 6:

[1290] The server applies optical character recognition (OCR) technology to the image file to extract text information from the image.

[1291] Step 7:

[1292] The server analyzes the text data and extracts sentence structure, keywords, and important points.

[1293] Step 8:

[1294] The server consolidates all the analyzed text data and organizes it into a single document.

[1295] Step 9:

[1296] The server automatically generates materials using the integrated text data according to a predefined template.

[1297] Step 10:

[1298] The server stores the generated materials in the user's personal or project folder.

[1299] Step 11:

[1300] The server notifies the user when the document is complete.

[1301] Step 12:

[1302] The user logs back into the system to view and download the completed document.

[1303] Step 13:

[1304] The user shares the generated materials with the relevant parties as needed.

[1305] Example 1

[1306] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1307] In today's business environment, there is a need to organize large amounts of data and create documents quickly and accurately. However, when dealing with data in various formats (video, audio, images, text), much of the processing is done manually, which requires time and effort, resulting in a decrease in work efficiency. Furthermore, errors are likely to occur during the data analysis and document creation process, resulting in inconsistent quality of the results. A system that can solve these problems and improve work efficiency and document quality is needed.

[1308] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1309] In this invention, the server includes: a means for a user to input data in the form of video, audio, image, or text; a means for the server to analyze the input data and extract character information using voice recognition technology and optical character recognition technology; a means for the server to automatically generate materials based on the extracted character information using a defined template; and a means for the server to store the generated materials and notify and distribute them to users. This makes it possible to effectively analyze and integrate various data formats, quickly and automatically generate high-quality materials, and significantly reduce the user's workload.

[1310] "User" means any person or entity that wishes to use the System to input data and generate materials.

[1311] "Server" refers to a computer system that receives data from users, analyzes it, and automatically generates materials.

[1312] "Data" means information, whether in the form of video, audio, images, or text, entered into the system by a user.

[1313] "Means for input" refers to the interface and functionality for users to upload data into the system.

[1314] "Analyzing" refers to the process in which the server uses voice recognition technology, optical character recognition technology, and the like to extract text information from the input data.

[1315] "Voice recognition technology" refers to the technology that analyzes voice data and converts the content into text format.

[1316] "Optical character recognition technology" refers to the technology that analyzes character information contained in image data and converts it into text format.

[1317] "Defined template" refers to a pre-set format or style used when automatically generating materials.

[1318] The "means for automatically generating materials" refers to a process in which the server creates materials using a template defined based on the extracted character information.

[1319] "Storing" refers to the server storing the generated material in its own storage device in order to provide it to the user.

[1320] "Distributing" refers to the server notifying the user of the generated material and making it accessible to the user.

[1321] This invention is a system that automatically generates materials based on video, audio, images, and text provided by users. The system consists of the following main components:

[1322] Data Entry Components

[1323] Users log in to the system's portal and use an interface to input the content they want to document in the form of video, audio, images, and text. Specifically, users select a file and click the upload button to send the data from their device to the server.

[1324] Data Analysis Components

[1325] The server analyzes the uploaded data. For example, for video and audio data, speech recognition technology (e.g., Google Speech-to-Text API) is used to convert the speech into text. For image data, optical character recognition technology (e.g., Google Cloud Vision API) is used to extract text information. The resulting text data is organized and integrated into a single text document.

[1326] Document generation component

[1327] The server automatically generates documents based on the analyzed text data according to predefined templates (e.g., Microsoft Word templates), for example, meeting minutes, business trip reports, knowledge sharing materials, etc.

[1328] Material storage and distribution component

[1329] The server saves the generated materials in the user's personal or project folder, and once the saving is complete, the server notifies the user and provides a download link for the materials, allowing the user to easily access the materials.

[1330] Specific processing flow example

[1331] 1. Example of data entry

[1332] If a user wants to upload a video of a presentation they gave on a business trip, they log in to the system, select "presentation.mp4," and click the upload button. The device sends this video file to the server, and the server sends a notification that it has been received. The user sees a message saying, "Upload completed."

[1333] 2. Specific examples of data analysis

[1334] The server analyzes "presentation.mp4" and converts the audio into text using the Google Speech-to-Text API. As a result of the analysis, text data such as "The main points of today's presentation are..." is extracted. Furthermore, text information such as "Figure 1: Sales Trends" is extracted from the slide images in the video using the Google Cloud Vision API, and all text data is integrated into a single document.

[1335] 3. Specific examples of data generation

[1336] The server generates a business trip report template based on the integrated text data. The content includes sections such as "Presentation Overview," "Main Points," and "Conclusion." Metadata such as "Date: October 10, 2023" and "Location: Tokyo" are added.

[1337] 4. Examples of material preservation and distribution

[1338] The generated business trip report is saved in the user's personal folder as "report20231010.docx." The server sends the user a notification that "The document has been generated and is available for download." The user then logs in to the system again, clicks the download link, and downloads the report. The downloaded report can be shared with team members via email.

[1339] Prompt Sentence Examples

[1340] For example, the following prompts can explain the process of uploading a video of a presentation a user gave on a business trip, converting the audio in the video into text using speech recognition technology, and automatically generating materials based on a template for a business trip report:

[1341] Simply upload a video of your business trip presentation to the system. We use speech recognition technology to convert the audio in the video into text, and automatically generate a business trip report template. The report will highlight key points and conclusions, and include metadata about the presentation date and the business trip destination.

[1342] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1343] Step 1: Data entry

[1344] Input: Users log in to the system's portal and select the video, audio, image, or text data they wish to document.

[1345] Specific operation: The user selects a file on the upload screen and clicks the upload button, at which point the device sends the selected file to the server.

[1346] Output: The server receives the file and sends a notification to the user that the upload is complete. The user sees the message "Upload completed."

[1347] Step 2: Receiving and storing data

[1348] Input: Files sent from the device (video, audio, image, text data).

[1349] Specific operation: The server stores the received file in temporary storage, which is used for subsequent analysis.

[1350] Output: The path and filename of the saved file. For example, / tmp / presentation.mp4.

[1351] Step 3: Data analysis (speech recognition)

[1352] Input: A video or audio file stored in temporary storage.

[1353] Specific operation: The server converts the voice data into text using speech recognition technology (e.g., Google Speech-to-Text API).

[1354] Data processing: Convert audio data in video or audio files into text format.

[1355] Output: The converted text data, e.g., "The main points of the presentation are...".

[1356] Step 4: Data Analysis (Optical Character Recognition)

[1357] Input: An image file or a frame from a video file saved in temporary storage.

[1358] Specific operation: The server uses optical character recognition technology (e.g., Google Cloud Vision API) to extract text information from the image.

[1359] Data processing: Converts text information in image files into text format.

[1360] Output: Extracted text data. For example, "Figure 1: Sales Trends."

[1361] Step 5: Data Integration

[1362] Input: Text data extracted using speech recognition and optical character recognition techniques.

[1363] Specific operation: The server organizes and integrates the multiple text data obtained into a single text document.

[1364] Data processing: Integrate scattered text data and organize it into documents.

[1365] Output: A consolidated text document, e.g., "Presentation Report Text Data".

[1366] Step 6: Generate materials

[1367] Input: A consolidated text document.

[1368] Specific operation: The server uses a predefined template to automatically generate documents based on the integrated text data, for example, using a Microsoft Word template.

[1369] Data processing: Place text documents according to templates and add required metadata (e.g. date, location, etc.).

[1370] Output: An automatically generated document file, for example, a business trip report (report20231010.docx).

[1371] Step 7: Preservation of materials and user notification

[1372] Input: Auto-generated documentation file.

[1373] Specific operation: The server saves the generated materials in the user's personal folder or project folder, and notifies the user when the saving is complete.

[1374] Output: The path and download link of the saved document, for example, / user / reports / report20231010.docx.

[1375] Step 8: Download and share materials

[1376] Input: The download link provided to the user.

[1377] Specific operation: The user logs in to the system again, clicks the download link to download the document, and then shares the document with relevant parties as needed.

[1378] Output: Downloaded documents and share them. For example, send a business trip report to a team member via email.

[1379] By following the above steps, this system automates the process from analyzing input data to generating, saving, and distributing materials, significantly reducing the user's workload.

[1380] (Application example 1)

[1381] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1382] Recording work processes and creating reports in factories requires a huge amount of time and effort when done manually, and there is a high risk of errors and information omissions. It also presents challenges in accurately understanding work content and quickly troubleshooting. Under these circumstances, automating work records and quickly generating reports is required to improve work efficiency and quality control throughout the factory.

[1383] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1384] In this invention, the server includes means for inputting video, audio, image, and text data, means for analyzing the input data and extracting character information, means for automatically generating materials using templates defined based on the extracted character information, means for saving and distributing the generated materials, and means for analyzing data captured by a factory work robot and automatically generating work reports. This automates the recording of work processes and the creation of reports in a factory, making it possible to improve work efficiency and strengthen quality control.

[1385] "Video" is data that records moving images or videos.

[1386] "Audio" is data that is a recording of a human voice or other sounds.

[1387] An "image" is data that records still visual information.

[1388] "Text" is information expressed in characters.

[1389] "Input means" refers to a method or device that allows a user or robot to provide video, audio, image, or text data to the system.

[1390] "Means of analysis" are techniques and methods for extracting useful information from the provided data.

[1391] "Text information" is information expressed as text.

[1392] "Extraction means" refers to the technology or equipment used to extract the necessary information from the data.

[1393] A "defined template" is a set of rules or formats for creating materials according to a specific format or structure.

[1394] "Means for automatically generating materials" refers to techniques or methods for automatically creating materials based on extracted information in accordance with predefined templates.

[1395] "Means for storing and distributing" refers to the techniques and methods for storing the automatically generated materials and distributing them to users who need them.

[1396] A "factory work robot" is a robotic device that automatically performs work in a factory.

[1397] A "work report" is a document that lists details of the work, results, problems, etc.

[1398] The following describes in detail an embodiment of the present invention. The present invention is a system that analyzes video, audio, and image data captured by a factory work robot and automatically generates a work report based on the data. The system of the present invention includes means for inputting video, audio, image, and text data, and means for analyzing each data and extracting text information. It also includes means for automatically generating materials using templates defined based on the extracted text information, and means for saving and distributing the generated materials. The specific configuration and operation of each means are described below.

[1399] First, the video, audio, and image data captured by the factory robot during work is sent to the system's server via input means, which include the robot's camera, microphone, and image capture device. The user then gives the robot appropriate instructions to collect data.

[1400] The server then analyzes the video, audio, and image data and extracts the necessary text information. This analysis uses the following techniques:

[1401] For voice recognition, a voice recognition engine (e.g., "Google Cloud Speech-to-Text") is used.

[1402] To extract text information from image data, an optical character recognition engine (e.g., "Tesseract OCR") is used.

[1403] A video analysis engine (e.g., OpenCV and deep learning models) is used to analyze the movements.

[1404] After data analysis, the extracted text information is automatically organized and consolidated by the server based on a predefined work report template, which includes, for example, the steps performed by the robot, any problems detected, and any suggested improvements.

[1405] Finally, the generated work report is stored by the server and distributed to the relevant parties as needed. The authenticity of the stored report may be verified by a digital signature, ensuring the reliability of the report content. Through this storage and distribution method, users and administrators can easily access and share the report.

[1406] A concrete example of this would be the following scenario. Suppose a robot is inspecting products on a factory production line. The robot captures video and audio during the inspection and sends the data to a server. The server analyzes this data and automatically generates a report containing the inspection results and any detected defects. This report becomes an important source of information for managers to view and take any necessary corrective measures promptly.

[1407] Examples of prompts using generative AI models include:

[1408] Generative AI prompts for work report generation systems:

[1409] 1. Please generate a work report based on the following text data.

[1410] 2. It should include a summary of the work, any issues found, and any suggested improvements.

[1411] 3. The output format is a Microsoft Word document.

[1412] In this way, the system of the present invention contributes to efficient management of factory work and improved quality.

[1413] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1414] Step 1:

[1415] Data collection by factory robots

[1416] input:

[1417] The user gives instructions to the factory robot to start work, and the robot collects video, audio, and image data using a camera, microphone, and image capture device.

[1418] output:

[1419] Data files for video (mp4 format), audio (wav format), and images (png or jpg format) are generated.

[1420] Specific behavior:

[1421] The robot's camera captures the work scene.

[1422] The microphone records your voice as you work.

[1423] An image capture device periodically takes still images.

[1424] Step 2:

[1425] Uploading data

[1426] input:

[1427] The video, audio, and image data collected by the robot is managed on a terminal.

[1428] output:

[1429] The data file is uploaded to the server.

[1430] Specific behavior:

[1431] The terminal receives data from the robot.

[1432] The device uploads the data to a server via the Internet.

[1433] Step 3:

[1434] Data analysis

[1435] input:

[1436] Video, audio, and image data uploaded to the server.

[1437] output:

[1438] Text data of character information and action information analyzed by the server.

[1439] Specific behavior:

[1440] The server analyzes the video data and extracts motion information for each specified frame.

[1441] The voice data is converted into text using a voice recognition engine (Google Cloud Speech-to-Text).

[1442] Text information is extracted from the image data using an optical character recognition engine (Tesseract OCR).

[1443] Step 4:

[1444] Material generation

[1445] input:

[1446] Text data of character information and action information analyzed by the server.

[1447] output:

[1448] Automatically generated work reports (Microsoft Word documents) based on predefined templates.

[1449] Specific behavior:

[1450] The server arranges the extracted text information according to a defined template.

[1451] Automatically generate work reports that include work details, issues detected, and suggested improvements.

[1452] Step 5:

[1453] Storage and distribution of materials

[1454] input:

[1455] Automatically generated work report (Microsoft Word document).

[1456] output:

[1457] Work reports stored in the server's cloud storage and distributed to stakeholders.

[1458] Specific behavior:

[1459] The server stores the generated work report in cloud storage.

[1460] Users can access the saved reports and distribute them to interested parties as needed.

[1461] Digital signature functionality is used to ensure the authenticity of reports.

[1462] The above steps create a system that automatically records factory work and creates reports, which improves work efficiency and strengthens quality control.

[1463] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1464] This invention combines a system that automatically generates materials based on user-provided video, audio, images, and text with an emotion engine that recognizes the user's emotions, enabling it to adjust the tone and content of the materials according to the user's emotional state. This system analyzes input data, extracts the user's emotional information, and automatically creates materials according to defined templates, significantly reducing the effort required to create materials and providing materials with more appropriate content.

[1465] System Overview

[1466] The system consists of the following major components:

[1467] 1. Data Entry Component

[1468] It provides an interface that allows users to input the content they want to document in the form of video, audio, images, and text.

[1469] Users log in to the system's portal and upload data.

[1470] 2. Data Analysis Component

[1471] The server analyzes the uploaded data, converting audio and video data into text using voice recognition technology, and extracting text information from image data using optical character recognition technology.

[1472] An emotion engine is used to recognize the user's emotions from video and audio, and organizes the emotional information together with text data.

[1473] 3. Document Generation Component

[1474] The server automatically generates materials based on the analyzed text data and emotional information according to predefined templates.

[1475] Adjust the tone and content of your materials depending on the user's emotional state.

[1476] 4. Material Storage and Distribution Component

[1477] The server stores the generated materials in the user's personal or project folder for easy access by the user.

[1478] Users can download the generated materials from within the system and share them with relevant parties as needed.

[1479] Specific processing flow

[1480] Data entry examples

[1481] For example, consider a case where a user wants to upload a recording of a meeting to the system. The user logs in to the system, selects the recording file, and clicks the upload button. The device then sends the recording file to the server and notifies the user when the upload is complete.

[1482] Specific examples of data analysis

[1483] Once the recording file is uploaded to the server, the server converts the voice data into text using speech recognition technology. At the same time, the emotion engine analyzes the tone, pitch, and speed of the voice to recognize the user's emotions. For example, if someone expresses strong interest or dissatisfaction during a meeting, that emotional information is recorded along with the text data.

[1484] Example of data generation

[1485] The analyzed text data and emotional information are organized and arranged according to a predefined meeting minutes template. The server automatically generates minutes based on the emotional information, emphasizing specific remarks and discussion points in an appropriate tone. For example, points that users showed strong interest in are recorded in particular detail.

[1486] Examples of material preservation and distribution

[1487] The automatically generated meeting minutes are saved in the user's personal folder by the server. Once saved, the user is notified and can download the minutes by logging back into the system. The downloaded minutes can then be distributed to relevant parties via email or the system's sharing function.

[1488] In this way, the present invention automates each process of document creation and further utilizes user emotional information to generate more accurate and appropriate documents, significantly improving work efficiency.

[1489] The processing flow will be explained below.

[1490] Step 1:

[1491] The user logs into the system's captive portal.

[1492] Step 2:

[1493] The user selects video, audio, image, or text data and clicks the upload button.

[1494] Step 3:

[1495] The terminal transmits the selected file to the server.

[1496] Step 4:

[1497] The server receives the uploaded data and identifies the data type (video, audio, image, text).

[1498] Step 5:

[1499] The server applies automatic speech recognition (ASR) technology to the audio and video files, converting the audio data into text.

[1500] Step 6:

[1501] The server applies optical character recognition (OCR) technology to the image file to extract text information from the image.

[1502] Step 7:

[1503] The server applies an emotion engine to the video or audio file to recognize the user's emotion. For example, the emotion engine analyzes the tone, pitch, and speed of the voice to identify whether the user is happy, angry, or surprised.

[1504] Step 8:

[1505] The server analyzes the text data and emotional information to extract sentence structure, keywords, and key points. The emotional information is used to emphasize and refine specific parts.

[1506] Step 9:

[1507] The server integrates all the analyzed text data and emotional information and organizes it into a single document.

[1508] Step 10:

[1509] The server automatically generates materials using the integrated text data and emotional information according to predefined templates, adjusting the tone and content of the materials by, for example, highlighting parts that express strong emotions.

[1510] Step 11:

[1511] The server stores the generated materials in the user's personal or project folder.

[1512] Step 12:

[1513] The server notifies the user when the document is complete.

[1514] Step 13:

[1515] The user logs back into the system to view and download the completed document.

[1516] Step 14:

[1517] Users can share the generated materials with stakeholders as needed by creating a sharing link within the system or sending it via email.

[1518] Example 2

[1519] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1520] Conventional document creation systems have been unable to generate documents that take the user's emotions into account, and as a result, the automatically generated documents often do not match the user's intentions or emotions. In addition, because many parts of the system rely on manual work, document creation takes time and effort.

[1521] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for a user to input video, audio, still image, and document data, means for analyzing the input data and extracting text information, means for recognizing the user's emotional information and adjusting the tone and content of the materials based on the emotional information, means for automatically generating materials using templates defined based on the extracted text information and emotional information, and means for saving and distributing the generated materials. This makes it possible to generate highly accurate materials that reflect the user's emotions.

[1522] "User" refers to any individual or organization that uses the System to input data and generate or download materials.

[1523] "Video" refers to digital or analog data that records moving images.

[1524] "Audio" refers to digital or analog data that records human speech or other sounds.

[1525] A "still image" refers to a single piece of image data that does not move.

[1526] "Document data" refers to data in the form of text files or documents that contain character information.

[1527] A "server" refers to a computer system that performs processes such as data analysis and automatic generation of materials.

[1528] "Template" refers to a defined format or layout used to automatically generate materials.

[1529] "Voice recognition technology" refers to the technology that analyzes voice data and converts it into text data.

[1530] "Optical character recognition technology (OCR)" refers to the technology of extracting character information from image data.

[1531] "Emotional information" refers to data obtained by analyzing the user's emotional state.

[1532] The "tone of a document" refers to the atmosphere and tone of the content and presentation of the document.

[1533] "Document generation" refers to the process of creating documents based on analyzed data.

[1534] "Preservation" refers to the keeping of generated materials in appropriate storage.

[1535] "Distribution" refers to providing generated materials to other users or interested parties.

[1536] This invention combines a system that automatically generates materials based on video, audio, still images, and text data provided by users with an emotion engine that recognizes the user's emotions, giving it the ability to adjust the tone and content of the materials according to the user's emotional state. This system analyzes the input data, extracts the user's emotional information, and automatically creates materials according to defined templates, significantly reducing the effort required to create materials and providing materials with more appropriate content.

[1537] System configuration

[1538] The system consists of the following major components:

[1539] 1. Data Entry Component

[1540] It provides an interface that allows users to input the content they want to document in the form of video, audio, still images, or documents.

[1541] Users log in to the system's portal and upload data.

[1542] 2. Data Analysis Component

[1543] The server analyzes the uploaded data, converting audio and video data into text using voice recognition technology, and extracting text information from still image data using optical character recognition (OCR).

[1544] An emotion engine is used to recognize the user's emotions from video and audio, and organizes the emotional information together with text data.

[1545] 3. Document Generation Component

[1546] The server automatically generates materials based on the analyzed text data and emotional information according to predefined templates.

[1547] Adjust the tone and content of your materials depending on the user's emotional state.

[1548] 4. Material Storage and Distribution Component

[1549] The server stores the generated materials in the user's personal or project folder for easy access by the user.

[1550] Users can download the generated materials from within the system and share them with relevant parties as needed.

[1551] Detailed component description

[1552] 1. Data Entry Component

[1553] Users first log in to the system's portal and upload the data they want to document (video, audio, still images, documents). The device then sends the selected files to the server and notifies the user when the upload is complete.

[1554] For example, if a user wants to upload a recording file of a meeting to the system, the user logs in, selects the recording file, and clicks the upload button. The device sends the recording file to the server and notifies the user when the upload is complete.

[1555] Prompt statement example (data entry):

[1556] A user logs in to the system, clicks the "Upload Data" button on the portal to upload the recording file of the meeting, selects the recording file and uploads it.

[1557] 2. Data Analysis Component

[1558] The server analyzes the uploaded data and uses speech recognition technology (e.g., speech recognition API) to convert audio and video data into text, and optical character recognition (OCR) technology (e.g., OCR software) to extract text information from still image data.

[1559] The server uses an emotion engine (e.g., emotion analysis API) to recognize the user's emotions from video and audio data, and organizes the emotional information together with the text data.

[1560] For example, after the server receives the uploaded recording file, it calls a speech recognition API to convert the voice data into text. At the same time, it uses an emotion analysis API to analyze the tone, pitch, speed, etc. of the voice to determine the user's emotion. For example, it generates a result such as "The user is angry" and stores it in a database along with the text data.

[1561] Example prompt (Data Analysis):

[1562] The server uses a speech recognition API to convert the voice data into text and an emotion analysis API to analyze the user's emotions.

[1563] 3. Document Generation Component

[1564] The server automatically generates materials based on the analyzed text data and emotion information according to a predefined template (e.g., document template).

[1565] The server generates materials based on the emotional information, emphasizing specific statements and discussion points in an appropriate tone. For example, parts that the user expressed strong interest in will be recorded in particular detail.

[1566] For example, the server reads the parsed text data and sentiment information, inserts these data into a template defined using the document template API, and the generated document is automatically saved as a "meeting minutes" file.

[1567] Example prompt (data generation):

[1568] The server organizes the analyzed text data and emotional information based on the document template API and automatically generates meeting minutes.

[1569] 4. Material Storage and Distribution Component

[1570] The server saves the generated materials in the user's personal folder or project folder (e.g., cloud storage service). Once the saving is complete, the user is notified.

[1571] Users can re-login to the system, access their personal folders, and download documents, and also share documents with relevant parties.

[1572] As a concrete example, the server uses a cloud storage API to save the generated "meeting minutes" file in the user's personal folder. After saving is complete, the server sends a notification to the user's device. The user then logs in again and downloads the document from their personal folder. The document can also be shared by clicking the "Share" button and entering the email addresses of those involved.

[1573] Example prompt (storage and distribution):

[1574] The server saves the automatically generated meeting minutes in cloud storage and notifies the user after saving is complete.

[1575] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1576] Step 1: Data entry

[1577] Specific explanation

[1578] Users log in to the system's portal and upload the data they want to document (video, audio, still images, documents). The terminal sends the selected files to the server and notifies the user when the upload is complete. The input is the file specified by the user, and the output is the data stored on the server.

[1579] Specific actions

[1580] After logging in, users can click the "Select File" button in the upload interface to select the file they want to document. By pressing the "Start Upload" button, the device will send the file to the server and display a progress bar. After the upload is complete, the device will display a notification message saying "Upload Completed."

[1581] Step 2: Data analysis

[1582] Specific explanation

[1583] The server analyzes the uploaded data. It uses voice recognition technology (e.g., voice recognition API) to convert audio and video data into text, and optical character recognition (OCR) technology (e.g., OCR software) to extract text information from still image data. It then uses an emotion engine (e.g., emotion analysis API) to recognize the user's emotions from the video and audio, and organizes the emotional information along with the text data. The input is the data stored on the server, and the output is the analyzed text data and emotional information.

[1584] Specific actions

[1585] After the server receives the uploaded audio file, it calls a speech recognition API to convert the audio data into text. At the same time, it uses an emotion analysis API to analyze the tone, pitch, speed, etc. of the voice to determine the user's emotion. For example, it generates a result such as "The user is angry" and stores it in the database along with the text data. For image files, it uses OCR software to extract characters and stores the analysis results in the database as text data.

[1586] Adding Inputs and Outputs

[1587] The input (data stored on the server) is audio files, image files, and document files, and the output (internal server data) is text data and emotional information obtained through analysis.

[1588] Step 3: Generate materials

[1589] Specific explanation

[1590] The server automatically generates materials based on the analyzed text data and emotional information according to a predefined template (e.g., a document template). The content of the materials is adjusted according to the user's emotional state. The input is the text data and emotional information stored on the server, and the output is the generated materials.

[1591] Specific actions

[1592] The server reads the parsed text data and emotion information and inserts this data into a template defined using the document template API. For example, it increases the font size or changes the color to highlight parts where the user expressed strong interest or anger. The generated document is saved as a "meeting minutes" file.

[1593] Adding Inputs and Outputs

[1594] The input (internal data of the server) is analyzed text data and emotion information, and the output (generated file) is material generated based on a template.

[1595] Step 4: Preservation and distribution

[1596] Specific explanation

[1597] The server saves the generated materials in the user's personal folder or project folder. Once saving is complete, a notification is sent to the user, who can then re-login to the system to download the materials. It also provides a function to share the materials with relevant parties. The input is the generated materials, and the output is the URL of the saved materials or a notification that saving has been completed.

[1598] Specific actions

[1599] The server uses the cloud storage API to save the generated "meeting minutes" file in the user's personal folder. After saving is complete, the server sends a notification to the user's device. The user can also log in to the system again, access their personal folder, and download the materials. The user can share the materials by clicking the "Share" button and entering the email addresses of the relevant parties.

[1600] Adding Inputs and Outputs

[1601] The input (the generated file) is the generated material, and the output (a user-accessible link or distributable data) is the URL of the material stored in cloud storage or a notification that the storage is complete.

[1602] (Application example 2)

[1603] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1604] Conventional document generation systems were capable of extracting textual information from user input data, but were unable to generate documents that reflected emotional information. As a result, users' intentions and nuances were sometimes not accurately conveyed, making it difficult to create more effective advertising proposals and documents. In particular, in marketing and advertising campaigns, proposals that capture the emotions of target users are required, so solving this problem is an urgent task.

[1605] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1606] In this invention, the server includes means for inputting video, audio, image, and text data, means for analyzing the input data and extracting character information and emotional information, means for automatically generating materials using templates defined based on the extracted character information and emotional information, and means for adjusting the tone and content of the materials based on the emotional information, and means for saving and distributing the generated materials. This enables the creation of more effective and detailed advertising proposal materials and materials that reflect the user's emotions.

[1607] "Motion picture" is a data format in which visual content is displayed continuously.

[1608] "Audio" is a data format that includes auditory information such as speech and sound effects.

[1609] An "image" is a data format for displaying fixed visual content.

[1610] "Text" is a data format that includes characters such as alphabets and kanji.

[1611] "Emotion information" is data that indicates the user's feelings and emotions, and is analyzed from the input data.

[1612] A "template" is a format used as a standard when generating materials.

[1613] "Materials" refers to documents or presentations created for a specific purpose or to organize information.

[1614] "Tone" refers to the overall atmosphere or mood of a document or piece of writing.

[1615] "Storage" refers to the act of recording generated materials in a database or file system.

[1616] "Distribution" refers to the act of sending or making available generated materials to other users or systems.

[1617] The present invention relates to a system for automatically generating proposal materials for an advertising campaign based on various input data. Hereinafter, an embodiment of the present invention will be described in detail.

[1618] First, the system provides an interface for inputting video, audio, image, and text data. Users log in to the system and upload this data, which is then sent to the server as the basis for creating materials. The server analyzes the input data and extracts text and emotional information. Audio data is converted into text using voice recognition technology, and image data is used to extract text information using optical character recognition technology.

[1619] At this time, emotional information is extracted from the analyzed audio and video using an emotion engine. This emotion engine analyzes the user's voice characteristics, such as tone, pitch, and speed, to recognize their emotional state. This emotional information and text information are then organized by the server.

[1620] The server then automatically generates ad campaign proposals using predefined templates based on the organized text and emotional information. The tone and content of the proposals are adjusted based on the user's emotional state. For example, specific ad copy or visuals may be emphasized based on areas of high interest.

[1621] The generated materials are stored on the server and made easily accessible to users, who can also download the materials from within the system and share them with interested parties.

[1622] The main hardware and software used in implementing the system are as follows:

[1623] Hardware: PC or smartphone with internet connection

[1624] Software: Python, cv2 library, speech_recognition library, transformers library

[1625] As a concrete example, consider a scenario in which a user wants to upload a recording of a meeting to a system. The user logs in to the system, selects the recording file, and clicks the upload button. The device then sends the recording file to the server. Once the upload is complete, the system converts the recording file into text using speech recognition technology, and automatically generates proposal materials for an advertising campaign based on the text data and emotional information.

[1626] An example prompt might look like this:

[1627] "Create an advertising campaign pitch from these voice memos. The memos include product features and highlights."

[1628] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1629] Step 1:

[1630] The user logs in to the system. The user enters authentication information on the provided login screen and accesses the system. This operation allows the user to specify which project to upload data to.

[1631] Step 2:

[1632] The user uploads video, audio, image, or text data. The user selects the data file on the upload screen and clicks the send button. The input data is sent from the device to the server. The input data format includes mp4, wav, jpg, pdf, txt, etc.

[1633] Step 3:

[1634] The server receives the uploaded data. The received data is first saved in a temporary location before being analyzed. For example, if audio data is provided in wav format, it will be saved in the specified directory.

[1635] Step 4:

[1636] The server analyzes the data and extracts text and emotional information. Specifically, it converts audio data into text using speech recognition technology, and extracts text from image data using optical character recognition (OCR). It then uses an emotion engine to analyze the tone, pitch, and speed of the audio and video to recognize the emotional state. In this step, audio or image data is received as input, and text and emotional information are generated as output.

[1637] Step 5:

[1638] The server automatically generates materials using templates defined based on text and emotional information. The server arranges the necessary data appropriately according to the template and adjusts the tone and content based on emotional information. For example, particularly detailed explanations are added to parts that show strong user interest. In this step, organized text and emotional information are received as input, and advertising campaign proposal materials are generated as output.

[1639] Step 6:

[1640] The server saves and distributes the generated materials. The automatically generated materials are saved in the user's personal folder or project folder. After saving is complete, a notification is sent to the user, who can then re-login to the system and download the materials. Furthermore, the user can share the downloaded materials with relevant parties via email or the sharing function. In this step, the generated materials are received as input, and saving and distribution are performed as output.

[1641] In this way, efficient and effective material creation is achieved through the specific processing steps of the system that automatically generates advertising campaign proposal materials that reflect user emotional information.

[1642] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1643] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1644] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1645] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1646] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1647] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1648] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1649] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1650] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1651] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1652] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1653] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1654] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1655] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1656] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1657] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1658] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1659] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1660] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1661] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1662] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1663] The following is further disclosed regarding the above embodiment.

[1664] (Claim 1)

[1665] A means of inputting video, audio, image, and text data;

[1666] A means for analyzing input data and extracting character information;

[1667] a means for automatically generating materials using a template defined based on the extracted character information;

[1668] a means of storing and distributing the generated materials;

[1669] A system including:

[1670] (Claim 2)

[1671] The system according to claim 1, further comprising means for converting the voice data into text using a voice recognition technique.

[1672] (Claim 3)

[1673] 10. The system of claim 1, further comprising means for extracting text information from the image data using optical character recognition techniques.

[1674] "Example 1"

[1675] (Claim 1)

[1676] a means for a user to input data in the form of video, audio, images, or text;

[1677] A means for the server to analyze the input data and extract text information using voice recognition technology and optical character recognition technology;

[1678] A means for automatically generating materials using a template defined by the server based on the extracted character information;

[1679] A server stores the generated materials and notifies and distributes them to users;

[1680] A system including:

[1681] (Claim 2)

[1682] 2. The system according to claim 1, further comprising means for the server to convert the video and audio data into text using voice recognition technology.

[1683] (Claim 3)

[1684] 10. The system of claim 1, further comprising means for the server to extract text information from the image data using optical character recognition techniques.

[1685] "Application Example 1"

[1686] (Claim 1)

[1687] A means of inputting video, audio, image, and text data;

[1688] A means for analyzing input data and extracting character information;

[1689] a means for automatically generating materials using a template defined based on the extracted character information;

[1690] a means of storing and distributing the generated materials;

[1691] A means for analyzing data captured by a factory work robot and automatically generating a work report;

[1692] A system including:

[1693] (Claim 2)

[1694] The system according to claim 1, further comprising means for converting the voice data into text using a voice recognition technique.

[1695] (Claim 3)

[1696] 10. The system of claim 1, further comprising means for extracting text information from the image data using optical character recognition techniques.

[1697] "Example 2: Combining Emotion Engines"

[1698] (Claim 1)

[1699] A means for a user to input video, audio, still images, and text data;

[1700] A means for analyzing input data and extracting character information;

[1701] means for recognizing a user's emotional information and adjusting the tone and content of the material based on the emotional information;

[1702] a means for automatically generating materials using a template defined based on the extracted character information and emotion information;

[1703] a means of storing and distributing the generated materials;

[1704] A system including:

[1705] (Claim 2)

[1706] The system according to claim 1, further comprising means for converting the voice data into text using a voice recognition technique.

[1707] (Claim 3)

[1708] 10. The system of claim 1, further comprising means for extracting text information from the still image data using optical character recognition techniques.

[1709] "Application example 2 when combining emotion engines"

[1710] (Claim 1)

[1711] A means of inputting video, audio, image, and text data;

[1712] means for analyzing input data and extracting character information and emotion information;

[1713] a means for automatically generating materials using templates defined based on the extracted character information and emotion information, and adjusting the tone and content of the materials based on the emotion information;

[1714] a means of storing and distributing the generated materials;

[1715] A system including:

[1716] (Claim 2)

[1717] The system according to claim 1, further comprising means for converting the voice data into text using a voice recognition technique.

[1718] (Claim 3)

[1719] 10. The system of claim 1, further comprising means for extracting text information from the image data using optical character recognition techniques. [Explanation of symbols]

[1720] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of inputting video, audio, image, and text data; A means for analyzing input data and extracting character information; a means for automatically generating materials using a template defined based on the extracted character information; a means of storing and distributing the generated materials; A system including:

2. The system according to claim 1 , further comprising means for converting the voice data into text using a voice recognition technique.

3. 10. The system of claim 1, further comprising means for extracting textual information from the image data using optical character recognition techniques.

Images