system

Abstract

Provide a system. A means for receiving an electronic file of reference material; means for converting the contents of the received electronic file into text data using optical character recognition technology; A method for analyzing text data using a natural language processing engine, extracting keywords and themes, and generating metadata. means for storing the metadata; means for receiving a search prompt and extracting relevant keywords from the prompt; A means for searching the database based on the extracted keywords and picking up related materials; and means for extracting important sections from the picked-up material and providing them to the user.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventionally, it has been difficult for users to efficiently search for reference materials and quickly find the appropriate materials. This difficulty is particularly pronounced when there are a large number of materials or when pinpointing specific information from the materials. Users must manually read the contents of the materials, and finding the appropriate material requires a great deal of effort and time. This often causes users great stress before they can find the information they are looking for. There is a need to provide a system that solves these problems and enables users to efficiently find the materials they need. [Means for solving the problem]

[0005] The present invention provides a system for efficiently searching electronic files of reference materials and quickly providing users with the information they desire. Specifically, the system includes a means for receiving electronic files of reference materials and a means for converting the contents of the received electronic files into text data using optical character recognition technology. The system also includes a means for analyzing the converted text data with a natural language processing engine, extracting keywords and themes, and generating metadata. The system further includes a database for storing the generated metadata and a means for receiving a search prompt and extracting relevant keywords from the prompt. The system then includes a means for searching the database based on the extracted keywords and selecting relevant materials. Finally, the system includes a means for extracting important sections from the selected materials and providing them to the user. This system allows users to efficiently find the information they desire, eliminating the labor and time required for conventional search methods.

[0006] A "reference" is a document or digital file used to provide specific information that a user may refer to depending on the purpose of use.

[0007] An "electronic file" is a document, slide, image, or other data format file stored in digital format.

[0008] "Means for receiving" refers to technology including interfaces and protocols for incorporating electronic files uploaded by users into the system.

[0009] "Optical character recognition technology" is a technology for analyzing character information in image files and converting it into text data.

[0010] "Text data" is written information stored in a digital format that is easy to analyze and search.

[0011] A "natural language processing engine" is a software system that analyzes text data and understands its meaning and structure.

[0012] "Keywords" are important words or phrases in the text data that represent the subject or content of the document.

[0013] A "theme" is a subject that indicates the main topic or concept of the entire document.

[0014] "Metadata" is information about the attributes and structure of data, and is used to make accessing and searching materials more efficient.

[0015] A "database" is a system for storing information in a structured digital format that allows for rapid retrieval and access.

[0016] A "search prompt" is a keyword or phrase that a user enters into the system that triggers a search for specific information.

[0017] "Extraction means" refers to techniques and methods for extracting relevant information and features from data.

[0018] A "search query" is a command or instruction to search a database for specific information.

[0019] "Pickup methods" are methods for selecting highly relevant materials from search results.

[0020] A "section" is a part of a document or slide, a divided part that contains a set of information or content.

[0021] The "delivery method" refers to the techniques and processes used to present selected information or materials to users.

[0022] A "thumbnail" is a reduced-size image of a document or slide that is displayed to allow easy viewing of the entire content. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0024] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0025] First, the terms used in the following description will be explained.

[0026] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0027] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0028] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0029] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0030] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0031] [First embodiment]

[0032] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0033] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0034] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0035] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0036] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0037] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0038] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0039] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0040] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0041] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0042] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0043] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0044] The present invention is a system for efficiently searching and suggesting reference materials, particularly those stored as electronic files, and automatically processing them to enable users to quickly and accurately obtain the information they desire.

[0045] This system is mainly composed of three entities: a server, a terminal, and a user. The specific operation of each is explained below.

[0046] Uploading and storing materials

[0047] User: First, the user accesses the system's upload screen, selects the electronic file (e.g., PDF, PPT, etc.) they want to use as reference, and clicks the upload button.

[0048] Terminal: Sends the file selected by the user to the server.

[0049] Server: The received file is stored in temporary storage. Then, using OCR (Optical Character Recognition) technology, the contents of the uploaded file are converted into text data. After conversion, each slide or page in the file is divided into sections and assigned a unique ID. This text data and section information are stored in a material database.

[0050] Text analysis and feature extraction of materials

[0051] Server: Analyzes text data stored in the resource database using a natural language processing (NLP) engine. Extracts key keywords and themes from titles, headings, and main text, and evaluates the importance of each section. The extracted keywords, themes, and importance information are stored in the resource database as metadata.

[0052] Entering and parsing search prompts

[0053] User: Enter the image or keywords of the desired material as a prompt.

[0054] Terminal: Sends the entered prompt to the server.

[0055] Server: Analyzes the received prompts, extracts relevant themes and keywords, and generates search queries based on the extracted themes and keywords.

[0056] Search and suggest related materials

[0057] Server: Searches the resource database using the generated search query. It compares the metadata in the resource database to select multiple highly relevant resources. It then extracts important slides and sections from each of the selected resources and generates summary information for each (title, keywords, importance, etc.). It then sends a list of the selected resources and key sections to the user's device.

[0058] Displaying the results

[0059] Terminal: Displays the user a list of proposal materials and the important slides and sections of each material. For each material, the title, summary, keywords, and thumbnails of important slides are displayed.

[0060] Users: Review the proposed materials and, if interested, view details or download them.

[0061] Specific examples

[0062] For example, consider the case where a user uploads a slide set for a conference presentation, "Research Presentation_Machine Learning.ppt." When the user selects a file using the upload screen and clicks the upload button, the device sends the file to the server. The server saves the received file in temporary storage and converts it into text data using OCR technology. The converted text data is divided into sections, and the information is stored in a materials database.

[0063] Next, when a user enters a prompt such as "latest applications of machine learning" into the search screen, the prompt is sent from the device to the server. The server analyzes the prompt and extracts keywords such as "machine learning," "latest," and "application." A search query is generated based on the extracted keywords, and by searching the document database, highly relevant documents are picked up.

[0064] Important slides (e.g., slides 3 and 7) are extracted from the selected materials, and summary information is generated for each. This information is sent to the user's device and displayed as a list of proposed materials.

[0065] Users can select the materials they want to know more about from the suggested list and view or download them, allowing them to obtain the information they are looking for quickly and efficiently.

[0066] Thus, the present invention is significantly more efficient than existing document retrieval systems and can quickly provide information that meets the user's needs.

[0067] The processing flow will be explained below.

[0068] Step 1: Upload your materials

[0069] User: The user accesses the system's upload screen, selects the electronic file (e.g., PDF, PPT, etc.) they want to use as reference, and clicks the upload button.

[0070] Step 2: Send the file

[0071] Terminal: Sends the file selected by the user to the server.

[0072] Step 3: Save the file

[0073] Server: Stores received files in temporary storage.

[0074] Step 4: OCR processing

[0075] Server: Launches an OCR (optical character recognition) engine and converts the contents of the uploaded file into text data.

[0076] Step 5: Sectioning

[0077] Server: Each slide or page in the text data file is divided into sections, and a unique ID is assigned to each section.

[0078] Step 6: Store in the database

[0079] Server: Stores the converted text data and section information in the material database.

[0080] Step 7: Natural Language Processing Analysis

[0081] Server: Analyzes the text data stored in the material database using a natural language processing (NLP) engine.

[0082] Step 8: Extracting keywords and themes

[0083] Server: Extracts key keywords and themes from titles, headings, and body content, and rates the importance of each section.

[0084] Step 9: Generate Metadata

[0085] Server: Generates extracted keywords, themes, and importance information as metadata and adds them to the material database.

[0086] Step 10: Enter the search prompt

[0087] User: Accesses the system's search screen and enters the image or keywords of the desired material as prompted.

[0088] Step 11: Sending prompts

[0089] Terminal: Sends the entered prompt to the server.

[0090] Step 12: Parse prompts

[0091] Server: Parses the received prompts and extracts relevant themes and keywords.

[0092] Step 13: Generate a search query

[0093] Server: Generates search queries based on the extracted themes and keywords.

[0094] Step 14: Search the documentation database

[0095] Server: Searches the resource database using the generated search query.

[0096] Step 15: Identify relevant materials

[0097] Server: Compares the metadata in the material database and selects multiple highly relevant materials.

[0098] Step 16: Extracting important sections

[0099] Server: Extracts key slides and sections from each document.

[0100] Step 17: Generate summary information

[0101] Server: Generates summary information (title, keywords, importance, etc.) for the extracted important slides and sections.

[0102] Step 18: Submit your proposal list

[0103] Server: Sends the list of proposal materials and the list of important sections to the user's terminal.

[0104] Step 19: View the results

[0105] Terminal: Displays the user a list of proposal materials and key slides and sections of each material.

[0106] Step 20: View details and download

[0107] Users: Review the suggested materials and select the ones that interest them for detailed viewing or download.

[0108] In this way, the system provides the user with the materials they require efficiently and quickly through a series of steps.

[0109] Example 1

[0110] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0111] Conventional reference material search systems have limitations in their ability to efficiently search and suggest the contents of electronic files, making it difficult for users to quickly and accurately obtain the information they need. Furthermore, they lacked the technology to improve the accuracy of suggesting highly relevant materials and the ability to extract and display important sections of materials. This resulted in the problem of users spending a great deal of time and effort finding the information they were looking for.

[0112] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0113] In this invention, the server includes: means for receiving electronic files of reference materials; means for converting the contents of the received electronic files into text data using optical character recognition technology; means for analyzing the text data with a natural language processing engine to extract keywords and themes and generate metadata; means for storing the metadata in a database; means for receiving search prompts and extracting relevant keywords from the prompts; means for searching the database based on the extracted keywords to select relevant materials; means for extracting important sections from the selected materials and providing them to the user; means for assigning unique IDs to the extracted sections and evaluating their importance; means for extracting themes from the prompts using a machine learning model; means for recommending highly relevant materials based on the metadata of the materials stored in the database; means for generating summary information of the relevant materials and transmitting it to the user terminal; and means for analyzing the search prompts and generating search queries. This allows users to quickly and efficiently obtain the information they need and effectively search for and suggest highly relevant materials.

[0114] "Electronic reference files" refers to materials and documents stored in digital formats such as PDF and PPT.

[0115] The "means for receiving" refers to a function for acquiring an electronic file when the server receives an uploaded file from a user.

[0116] "Optical character recognition technology" is a technology called OCR (Optical Character Recognition), which analyzes characters in an image and converts them into text data.

[0117] "Text data" refers to data in which only character information is extracted from the contents of an electronic file and saved in text format.

[0118] A "natural language processing engine" is a computer science technology for analyzing text data to extract keywords and themes.

[0119] "Metadata" refers to data that includes keywords, themes, section importance information, and the like extracted from text data.

[0120] A "database" is a system for efficiently storing, retrieving, and managing various types of data.

[0121] A "search prompt" is a keyword or phrase that a user enters to search for the information they need.

[0122] "Means for extracting relevant keywords" refers to a function for extracting important keywords from a search prompt.

[0123] A "search query" is a specific search condition generated to search a database.

[0124] "Means for picking out relevant materials" refers to a function for selecting relevant materials from a database based on a search query.

[0125] "Means for extracting important sections" refers to a function for selecting particularly important parts from the picked-up material.

[0126] A "unique ID" is an identifier assigned to each section to uniquely identify it.

[0127] The "means for evaluating the importance" refers to a function for quantitatively evaluating the importance of the extracted section.

[0128] A "machine learning model" is an algorithm or technique that learns from data and makes predictions or classifications based on that data.

[0129] "Means for recommending highly relevant materials" refers to a function for selecting and suggesting materials that best fit the user's search prompt.

[0130] The "means for generating summary information and transmitting it to the user terminal" refers to a function for generating a summary of the picked-up material and transmitting it to the user terminal.

[0131] The present invention is a system for efficiently searching and suggesting reference materials. This system automatically processes materials stored as electronic files, enabling users to quickly and accurately obtain the information they desire. The system is primarily composed of three entities: a server, a terminal, and a user. The specific operation of each is described below.

[0132] Uploading and storing materials

[0133] The user accesses the system's upload screen, selects the electronic file they wish to use as reference (e.g., PDF, PPT, etc.), and clicks the upload button. The terminal sends the file selected by the user to the server. The server stores the received file in temporary storage. The server then uses OCR (optical character recognition) technology to convert the contents of the uploaded file into text data. Tesseract OCR is used for the conversion. After conversion, each slide or page in the file is divided into sections and assigned a unique ID. This text data and section information are stored in a material database. The material database uses MySQL (registered trademark).

[0134] Text analysis and feature extraction of materials

[0135] The server analyzes the text data stored in the document database using a natural language processing (NLP) engine. The NLP engine used is SpaCy. It extracts key keywords and themes from titles, headings, and body content, and uses the TF-IDF algorithm to evaluate the importance of each section. The extracted keywords, themes, and importance information are stored in the document database as metadata.

[0136] Entering and parsing search prompts

[0137] The user enters an image of the desired material or keywords as a prompt into the search screen. The device sends the entered prompt to the server. The server analyzes the received prompt and extracts related themes and keywords. The BERT model is used to analyze the prompt. A search query is generated based on the extracted keywords.

[0138] Search and suggest related materials

[0139] The server searches the document database using the generated search query. It compares the results with the metadata in the document database to select multiple highly relevant documents. It calculates a relevance score and selects the top documents. It extracts the most important slides and sections from each of the selected documents and generates summary information (title, keywords, importance, etc.) for each. The generated summary information is sent from the server to the user's device in JSON format.

[0140] Displaying the results

[0141] The terminal displays a list of proposed materials and the important slides and sections of each material to the user. For each material, the title, summary, keywords, and thumbnails of important slides are displayed. HTML / CSS and JavaScript (registered trademark) are used to display the user interface. The user can check the proposed materials and, if interested, can view details or download them. Downloads are performed using an HTTP GET request.

[0142] Specific examples

[0143] For example, consider the case where a user uploads a slide set for a conference presentation, "Research Presentation_Machine Learning.ppt." When the user selects a file using the upload screen and clicks the upload button, the device sends the file to the server. The server saves the received file in temporary storage and converts it into text data using OCR technology. The converted text data is divided into sections, and the information is stored in a materials database.

[0144] Next, if a user enters a prompt such as "latest applications of machine learning" into the search screen, the prompt is sent from the device to the server. The server analyzes the prompt and extracts keywords such as "machine learning," "latest," and "application." A search query is generated based on the extracted keywords, and highly relevant materials are picked out by searching the materials database. Important slides (e.g., slides 3 and 7) are extracted from the picked materials, and summary information is generated for each. This information is sent to the user's device and displayed as a list of suggested materials.

[0145] This system allows users to quickly and efficiently obtain the information they need, significantly improving the efficiency of research and work. As a result, it becomes possible to provide information that meets the user's needs appropriately.

[0146] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0147] Step 1:

[0148] The user accesses the upload screen of the system, selects the electronic file they want to use as reference, and clicks the upload button.

[0149] Input: An electronic file selected by the user (e.g. PDF, PPT, etc.)

[0150] Output: Request to upload selected files

[0151] Step 2:

[0152] The terminal transmits the file selected by the user to the server.

[0153] Input: User upload request and selected electronic file

[0154] Output: A request for an electronic file to be sent to the server

[0155] Step 3:

[0156] The server stores the received file in temporary storage.

[0157] Input: Electronic file sent from the terminal

[0158] Output: Electronic file saved in temporary storage

[0159] Step 4:

[0160] The server uses OCR technology to convert the contents of the uploaded file into text data.

[0161] Input: Electronic files stored in temporary storage

[0162] Output: Converted text data

[0163] Step 5:

[0164] The server divides the text data into sections and assigns each a unique ID.

[0165] Input: Converted text data

[0166] Output: Text data with a unique ID assigned to each section

[0167] Step 6:

[0168] The server stores the section information and text data in a material database.

[0169] Input: Text data divided into sections

[0170] Output: Text data stored in the material database and its section information

[0171] Step 7:

[0172] The server uses a natural language processing engine to analyze the text data stored in the document database, extract key keywords and themes, and evaluate the importance of each section.

[0173] Input: Text data stored in the material database

[0174] Output: Extracted keywords, themes, and importance of each section

[0175] Step 8:

[0176] The server stores the extracted keywords, themes, and importance information as metadata in the document database.

[0177] Input: Extracted keywords, themes, and importance information

[0178] Output: Metadata stored in the material database

[0179] Step 9:

[0180] The user enters an image of the material they want or keywords into the search screen as a prompt.

[0181] Input: Search prompt

[0182] Output: User search request

[0183] Step 10:

[0184] The terminal sends the entered prompt to the server.

[0185] Input: User search request

[0186] Output: Sending a search prompt to the server

[0187] Step 11:

[0188] The server analyzes the prompts it receives and uses machine learning models to extract relevant themes and keywords.

[0189] Input: Submitted search prompt

[0190] Output: Themes and keywords extracted from the prompt

[0191] Step 12:

[0192] The server generates a search query based on the extracted themes and keywords.

[0193] Input: Extracted themes and keywords

[0194] Output: Generated search query

[0195] Step 13:

[0196] The server searches the document database using the generated search query and picks out multiple highly relevant documents.

[0197] Input: Generated search query

[0198] Output: Related materials picked up

[0199] Step 14:

[0200] The server extracts the most important slides and sections from each document and generates a summary.

[0201] Input: Picked up materials

[0202] Output: Summary information (title, keywords, importance, etc.)

[0203] Step 15:

[0204] The server transmits a list of proposal materials including summary information to the user's terminal.

[0205] Input: Summary Information

[0206] Output: A list of proposal materials sent to the user's device

[0207] Step 16:

[0208] The device displays a list of proposal materials and important slides and sections of each material to the user.

[0209] Input: Proposal list sent from the server

[0210] Output: A list of proposals displayed to the user

[0211] Step 17:

[0212] The user can review the proposed materials and, if interested, view them in detail or download them.

[0213] Input: Displayed proposal list

[0214] Output: View detailed view or download document

[0215] Through the above steps, the user can quickly and efficiently obtain the information he or she needs.

[0216] (Application example 1)

[0217] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0218] Factories and industrial sites often contain vast amounts of technical documents and manuals, but it is not easy to quickly and accurately search through them and obtain the information needed. Managing and searching through these documents takes a great deal of time and effort, especially when there are many paper documents and printed materials. This situation reduces work efficiency and impacts productivity. Furthermore, delays in accessing necessary information can lead to delays in equipment maintenance and troubleshooting. To address these challenges, an effective document management and search system is needed.

[0219] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0220] In this invention, the server includes: means for receiving electronic files of reference materials; means for converting the contents of the received electronic files into text data using optical character recognition technology; means for analyzing the text data with a natural language processing engine to extract keywords and themes and generate metadata; means for storing a database for storing the metadata; means for receiving search prompts and extracting relevant keywords from the prompts; means for searching a database based on the extracted keywords to find relevant materials; means for extracting important sections from the found materials and providing them to a user; means for a user to scan paper documents or printed materials with a smartphone and upload the images; means for processing the uploaded images with OCR technology and converting them to text; and means for searching a database using a TF-IDF-based search algorithm to suggest relevant materials in response to a user's search query, enabling factories and industrial sites to quickly and accurately search for necessary materials and provide them to users.

[0221] "Electronic files of reference materials" are technical documents and manuals related to work in factories and industrial sites that are stored in digital format.

[0222] Optical character recognition (OCR) is a technology that scans characters written on paper documents or printed materials and converts them into digital character data.

[0223] "Text data" refers to the text information in electronic files converted using OCR technology and stored in digital format.

[0224] A "natural language processing engine (NLP)" is a program that analyzes text data and automatically extracts important keywords and themes.

[0225] "Metadata" refers to additional information extracted from text data, such as keywords, themes, and importance, which is used to efficiently search and organize data within a database.

[0226] A "database" is a system for centrally managing and storing metadata and related material information.

[0227] A "search prompt" is a keyword or query that a user enters to specifically describe the information they are seeking.

[0228] A "TF-IDF-based search algorithm" is a statistical method for evaluating the importance of documents in a database based on a search query and deriving highly relevant materials.

[0229] "Scanning with a smartphone" refers to the process by which a user uses the smartphone's camera function to convert paper documents or printed materials into digital images.

[0230] "Uploading" is the act of a user transferring a digital image or electronic file to a cloud system such as a server.

[0231] "Important sections" are those parts of the selected material that are particularly relevant and useful to the user.

[0232] The present invention is a document management and search system for use in factories and industrial sites. This system is composed of three main components: a server, a terminal, and a user. The specific operation of each component will be explained below.

[0233] Uploading and storing materials

[0234] User: The user uses their smartphone to scan paper documents or printed materials and convert them into digital images, then selects the scanned image on the application's upload screen and clicks the upload button.

[0235] Terminal: Sends the image file selected by the user to the server.

[0236] Server: The received image files are stored in temporary storage and the contents are converted into text data using OCR (Optical Character Recognition) technology. After conversion, each file is divided into sections and assigned a unique ID. This text data and section information are stored in a material database.

[0237] Text analysis and feature extraction of materials

[0238] Server: Analyzes text data stored in the resource database using a natural language processing (NLP) engine. Extracts key keywords and themes and evaluates the importance of each section. The extracted keywords, themes, and importance information are stored in the resource database as metadata.

[0239] Entering and parsing search prompts

[0240] User: The user enters a search prompt into the application for the required material, for example, "Basic steps in PLC programming."

[0241] Terminal: Sends the entered prompt to the server.

[0242] Server: Analyzes the received prompts, extracts relevant themes and keywords, and generates search queries based on the extracted themes and keywords.

[0243] Search and suggest related materials

[0244] Server: Searches the document database using the generated search query. It uses a TF-IDF-based search algorithm to evaluate the importance of documents in the database and selects highly relevant documents. It then extracts important slides and sections from each of the selected documents and generates summary information for each (title, keywords, importance, etc.). It then sends a list of the selected documents and key sections to the user's device.

[0245] Displaying the results

[0246] Terminal: Displays the user a list of proposed materials and the important sections of each material. For each material, displays the title, summary, keywords, and thumbnails of important sections.

[0247] Users: Review the proposed materials and, if interested, view details or download them.

[0248] The system's hardware mainly requires a smartphone with a camera and a high-performance server. It uses the pytesseract library for OCR technology, and the nltk and scikit-learn libraries for natural language processing. The TF-IDF algorithm is used for data analysis of the documents.

[0249] Specific examples

[0250] For example, consider a user on a factory floor looking for a "Troubleshooting Guide for PLC Programming." The user scans a paper manual with their smartphone and uploads it to the server using an application. Next, they input "Basic Steps for PLC Programming" as a search prompt. The server analyzes this prompt and uses a TF-IDF-based algorithm to search and suggest relevant documents from the database. The user then checks the detailed view of the displayed document list to obtain the information they need.

[0251] The above method makes it possible to quickly and accurately search for and provide users with the necessary documents and information at factories and industrial sites. This system not only improves work efficiency at the site, but also speeds up troubleshooting and maintenance work by providing timely and appropriate information.

[0252] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0253] Step 1:

[0254] A user uses a smartphone to scan paper documents or printed materials and convert them into digital images. Then, they select the scanned image on the application's upload screen and click the upload button. At this point, the input is the scanned image file, and the output is an upload request.

[0255] Step 2:

[0256] The terminal sends the image file selected by the user to the server. The input here is the scanned image file, and the output is the image file sent to the server. This sending process is performed via the Internet.

[0257] Step 3:

[0258] The server stores the received image file in temporary storage. It then uses OCR technology to convert the image file into text data. The input here is the sent image file, and the output is text data. Specifically, it uses the pytesseract library to recognize the text in the image and convert it into text.

[0259] Step 4:

[0260] The server divides the converted text data into sections, assigns unique IDs, and stores them in a database. In this step, the input is text data, and the output is text data in a database divided into sections.

[0261] Step 5:

[0262] The server uses a natural language processing engine (NLP) to analyze the text data stored in the resource database and extract key keywords and themes. The input is the text data in the database, and the output is metadata. Specifically, it uses NLP technology to extract important words and phrases from the text and stores them as metadata.

[0263] Step 6:

[0264] The user enters a search prompt for the required material on the application search screen, for example, a keyword such as "Basic steps of PLC programming." The input of this step is the search prompt, and the output is a prompt transmission request from the terminal to the server.

[0265] Step 7:

[0266] The terminal sends the entered prompt to the server, where the input is the search prompt entered by the user and the output is a prompt transmission request to the server.

[0267] Step 8:

[0268] The server analyzes the prompts it receives, extracts relevant themes and keywords, and generates a search query based on that information. The input is the prompt, and the output is the generated search query. This analysis is performed using a generative AI model.

[0269] Step 9:

[0270] The server uses the generated search query to search the resource database, where the input is the search query and the output is a list of relevant resources. A TF-IDF-based search algorithm is used to evaluate and select relevant documents.

[0271] Step 10:

[0272] The server extracts important sections from each of the documents picked up and generates summary information (title, keywords, importance, etc.) for each. The input is a list of documents, and the output is a list of proposed documents with summary information.

[0273] Step 11:

[0274] The server sends the list of proposal materials and information on important sections of each material to the user's terminal. The input is the list of proposal materials with summary information, and the output is a transmission request to the user's terminal.

[0275] Step 12:

[0276] The terminal displays the list of proposal materials and important sections of each material to the user. The input is the list of proposal materials sent from the server, and the output is the material information displayed to the user.

[0277] Step 13:

[0278] The user checks the proposed materials and, if interested, can view details or download them. The input is the material information displayed to the user, and the output is the downloaded detailed materials.

[0279] This series of processes makes it possible to quickly and accurately search for necessary materials and information at factories and industrial sites and provide them to users.

[0280] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0281] The present invention is a system for efficiently searching electronic files of reference materials and quickly providing users with the information they desire. Furthermore, the present invention has the ability to recognize a user's emotions and customize search results based on those emotions. This system is specifically designed to automatically process materials stored as electronic files, enabling users to quickly and accurately obtain the information they desire.

[0282] This system is mainly composed of three entities: a server, a terminal, and a user. The specific operation of each is explained below.

[0283] Uploading and storing materials

[0284] User: The user accesses the system's upload screen, selects the electronic file (e.g., PDF, PPT, etc.) they want to use as reference, and clicks the upload button.

[0285] Terminal: Sends the file selected by the user to the server.

[0286] Server: The received file is stored in temporary storage. Then, using OCR (Optical Character Recognition) technology, the contents of the uploaded file are converted into text data. After conversion, each slide or page in the file is divided into sections and assigned a unique ID. This text data and section information are stored in a material database.

[0287] Text analysis and feature extraction of materials

[0288] Server: Analyzes text data stored in the resource database using a natural language processing (NLP) engine. Extracts key keywords and themes from titles, headings, and main text, and evaluates the importance of each section. The extracted keywords, themes, and importance information are stored in the resource database as metadata.

[0289] Emotion recognition by emotion engine

[0290] On the device: When a user searches, the emotion engine collects the user's voice data or facial image data.

[0291] Server: The emotion engine analyzes the collected data and recognizes the user's emotional state, for example, identifying whether the user is anxious or calm.

[0292] Entering and parsing search prompts

[0293] User: Enter the image or keywords of the desired material as a prompt.

[0294] Terminal: Sends the entered prompt to the server.

[0295] Server: Analyzes the received prompts, extracts relevant themes and keywords, and generates search queries based on the extracted themes and keywords.

[0296] Search and suggest related materials

[0297] Server: Searches the resource database using the generated search query, compares it with the resource database metadata, and selects multiple highly relevant resources.

[0298] Server: Extracts important slides and sections from each presentation and customizes the order and content of the presentation based on the user's perceived emotions. For example, prioritize short, concise presentations for impatient users.

[0299] Displaying the results

[0300] Terminal: Presents users with a customized list of presentation materials and key slides and sections for each material. For each presentation, it displays the title, summary, keywords, and thumbnails of key slides.

[0301] Users: Review the proposed materials and, if interested, view details or download them.

[0302] Specific examples

[0303] For example, consider the case where a user uploads a slide set for a conference presentation, "Research Presentation_Machine Learning.ppt." When the user selects a file using the upload screen and clicks the upload button, the device sends the file to the server. The server saves the received file in temporary storage and converts it into text data using OCR technology. The converted text data is divided into sections, and the information is stored in a materials database.

[0304] Next, when a user enters a prompt such as "latest applications of machine learning" into the search screen, the prompt is sent from the device to the server. The server analyzes the prompt and extracts keywords such as "machine learning," "latest," and "application." A search query is generated based on the extracted keywords, and by searching the document database, highly relevant documents are picked up.

[0305] At that time, the device sends the user's voice data and facial image data to the emotion engine, which then recognizes the user's emotions.The server then customizes the presentation order and content of the selected materials based on that emotional information.For example, if the user is feeling impatient, it will prioritize displaying materials that summarize only the main points.

[0306] Finally, the list of suggested materials is sent to the terminal, where the user can review the list and view or download the materials they are interested in. In this way, the system provides the most appropriate materials according to the user's emotional state, enabling the user to quickly and efficiently obtain the information they are looking for.

[0307] The processing flow will be explained below.

[0308] Step 1: Upload your materials

[0309] User: The user accesses the system's upload screen, selects the electronic file (e.g., PDF, PPT, etc.) they want to use as reference, and clicks the upload button.

[0310] Step 2: Send the file

[0311] Terminal: Sends the file selected by the user to the server.

[0312] Step 3: Save the file

[0313] Server: Stores received files in temporary storage.

[0314] Step 4: OCR processing

[0315] Server: Launches an OCR (optical character recognition) engine and converts the contents of the uploaded file into text data.

[0316] Step 5: Sectioning

[0317] Server: Each slide or page in the text data file is divided into sections, and a unique ID is assigned to each section.

[0318] Step 6: Store in the database

[0319] Server: Stores the converted text data and section information in the material database.

[0320] Step 7: Natural Language Processing Analysis

[0321] Server: Analyzes the text data stored in the material database using a natural language processing (NLP) engine.

[0322] Step 8: Extracting keywords and themes

[0323] Server: Extracts key keywords and themes from titles, headings, and body content, and rates the importance of each section.

[0324] Step 9: Generate Metadata

[0325] Server: Generates extracted keywords, themes, and importance information as metadata and adds them to the material database.

[0326] Step 10: Enter the search prompt

[0327] User: Accesses the system's search screen and enters the image or keywords of the desired material as prompted.

[0328] Step 11: Sending prompts

[0329] Terminal: Sends the entered prompt to the server.

[0330] Step 12: Parse prompts

[0331] Server: Parses the received prompts and extracts relevant themes and keywords.

[0332] Step 13: Generate a search query

[0333] Server: Generates search queries based on the extracted themes and keywords.

[0334] Step 14: Collect emotion data

[0335] Device: In parallel with entering the search prompt, the user's voice data and facial image data are sent to the emotion engine.

[0336] Step 15: Analyze the emotion data

[0337] Server: The emotion engine analyzes the collected voice data and facial image data to recognize the user's emotional state, for example, determining whether they are "anxious" or "calm."

[0338] Step 16: Search the documentation database

[0339] Server: Searches the resource database using the generated search query.

[0340] Step 17: Identify relevant materials

[0341] Server: Compares the metadata in the material database and selects multiple highly relevant materials.

[0342] Step 18: Extracting important sections

[0343] Server: Extracts key slides and sections from each document.

[0344] Step 19: Emotional Customization

[0345] Server: Customize the presentation order and content of the selected materials based on the user's recognized emotional state. For example, prioritize displaying materials that summarize the main points to a user who is feeling impatient.

[0346] Step 20: Submit your proposal list

[0347] Server: Sends a customized list of proposal materials and a list of important sections to the user's device.

[0348] Step 21: View the results

[0349] Terminal: Displays the user a list of proposal materials and key slides and sections of each material.

[0350] Step 22: View details and download

[0351] Users: Review the suggested materials and select those that interest them for detailed viewing or download.

[0352] Example 2

[0353] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0354] Conventional document search systems have difficulty in quickly and accurately providing the information users are looking for. Furthermore, because they provide uniform search results without considering the user's emotions or state, it is difficult to provide information that meets the user's needs. Furthermore, there are often too many search results, which can lead to important information being overlooked, resulting in a decrease in user satisfaction.

[0355] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0356] In this invention, the server includes means for receiving electronic files of reference materials, means for converting the contents of the received electronic files into text data using optical character recognition technology, means for analyzing the text data with a natural language processing engine to extract keywords and themes and generate metadata, means for analyzing audio data or image data and recognizing the user's emotions, means for searching a database based on the extracted keywords and the recognized emotions to select relevant materials, and means for extracting important sections from the selected materials and providing them to the user. This makes it possible to provide appropriate information according to the user's emotional state, enabling the user to quickly and accurately obtain the information they are looking for.

[0357] "Reference materials" refers to electronic files that users use to obtain information, such as PDFs and PPTs.

[0358] "Optical character recognition technology" is a technology that reads characters from image data and converts them into text data.

[0359] "Text data" refers to data that contains only text information and that has been converted using optical character recognition technology.

[0360] "Natural language processing engine" is a general term for software and algorithms that analyze text data and extract keywords and themes.

[0361] "Metadata" refers to data attribute information, such as keywords, themes, and relevance evaluation information extracted from text data.

[0362] A "search prompt" refers to the keywords or phrases that a user enters about the search target.

[0363] "Emotion recognition" is a technology that analyzes voice data and image data to identify a user's emotional state.

[0364] "Database" refers to an information system that stores data in a structured way to facilitate retrieval and access.

[0365] "Key sections" are those parts of the document that are assessed as being particularly important and include key points and summary information.

[0366] A "thumbnail" is a small version of a document or image that allows you to easily check the content.

[0367] A "machine learning model" refers to an algorithm or model that learns patterns based on data and makes predictions and classifications.

[0368] The present invention is a system for efficiently searching electronic files of reference materials and quickly providing users with the information they desire. Furthermore, the present invention has the ability to recognize users' emotions and customize search results based on those emotions. This system is specifically designed to automatically process materials stored as electronic files, enabling users to quickly and accurately obtain the information they desire.

[0369] This system is mainly composed of three entities: a server, a terminal, and a user. The specific operations performed by each entity are shown below.

[0370] Server Functions and Operations

[0371] The server performs processing using the following hardware and software.

[0372] Hardware: High-performance processor, memory, and storage

[0373] Software: OCR technology (e.g., Tesseract), natural language processing engines (e.g., spaCy, NLTK), emotion recognition engines (e.g., Microsoft® Azure® Cognitive Services, OpenCV)

[0374] The server first receives the electronic file uploaded by the user and stores it in temporary storage. Next, it uses OCR technology to convert the contents of the electronic file into text data. After conversion, each slide or page in the file is divided into sections and assigned a unique ID. This text data and section information are then stored in a material database.

[0375] The text data stored in the resource database is analyzed using a natural language processing engine. Key keywords and themes are extracted from the title, headings, and body of the text, and the importance of each section is evaluated. The extracted keywords, themes, and importance information are saved in the resource database as metadata.

[0376] The emotion engine analyzes the voice data or facial image data collected when the user searches to recognize the user's emotional state, which is also stored in the database.

[0377] The server analyzes the search prompt sent by the user and extracts relevant themes and keywords using a generative AI model (e.g., BERT, GPT-3 (registered trademark)). It then generates a search query based on the extracted themes and keywords and searches the document database.

[0378] The generated search query is used to search the document database to select multiple highly relevant documents, extract important slides and sections from each document, and customize the order and content of the documents to be presented based on the user's perceived emotions.

[0379] Device features and operations

[0380] A terminal is a device that a user uses as an interface, and performs processing using the following hardware and software.

[0381] Hardware: microphone, camera, display

[0382] Software: Web browser, communication module

[0383] When a user uploads a document, the terminal transmits the electronic file selected by the user to the server. When a user enters a search prompt, the terminal also collects the user's voice data and facial image data and transmits them to the server.

[0384] User operations

[0385] Users access the system's upload screen, select the electronic file they want to use as reference, and click the upload button. They also enter keywords related to the information they are looking for as prompted on the search screen. They can check the list of suggested materials displayed on the terminal and view or download the materials they are interested in in detail.

[0386] Specific examples

[0387] For example, consider the case where a user uploads a slide set for a conference presentation, "Research Presentation_Machine Learning.ppt." When the user selects a file using the upload screen and clicks the upload button, the device sends the file to the server. The server saves the received file in temporary storage and converts it into text data using OCR technology. The converted text data is divided into sections, and the information is stored in a materials database.

[0388] Next, when a user enters a prompt such as "latest applications of machine learning" into the search screen, the prompt is sent from the device to the server. The server analyzes the prompt and extracts keywords such as "machine learning," "latest," and "application." A search query is generated based on the extracted keywords, and by searching the document database, highly relevant documents are picked up.

[0389] At that time, the device sends the user's voice data and facial image data to the emotion engine, which then recognizes the user's emotions.The server then customizes the presentation order and content of the selected materials based on that emotional information.For example, if the user is feeling impatient, it will prioritize displaying materials that summarize only the main points.

[0390] Finally, the list of suggested materials is sent to the terminal, and the user can check the list and view or download the materials of interest in detail.

[0391] Examples of prompt statements

[0392] The prompt to be input to the generative AI model is as follows:

[0393] "I'm looking for the latest papers related to my research topic."

[0394] "What's the latest news on automotive battery technology?"

[0395] "I'd like to know what to do about this week's economic trends."

[0396] In this way, the system allows users to quickly and efficiently obtain the information they desire.

[0397] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0398] Step 1: Upload and store your materials

[0399] Input: An electronic file selected by the user (e.g. PDF, PPT, etc.)

[0400] Output: Text data and section information are stored in a resource database.

[0401] Specific behavior:

[0402] User: The user accesses the system's upload screen, selects the electronic file they wish to upload, and clicks the upload button.

[0403] Terminal: Sends the file selected by the user to the server. The terminal encodes the file as binary data and sends it via an HTTP request.

[0404] Server: The server stores the received files in temporary storage. It then uses Tesseract OCR technology to convert the file contents into text data. The converted text data is divided into sections for each slide or page, and a unique ID is assigned. This text data and section information are stored in a material database.

[0405] Step 2: Text analysis and feature extraction of the material

[0406] Input: Text data stored in the resource database

[0407] Output: Generates metadata for keywords, themes, and importance information

[0408] Specific behavior:

[0409] Server: The server analyzes the text data stored in the document database using a natural language processing (NLP) engine. Software used includes spaCy and the NLTK library. After analysis, it extracts key keywords and themes from the title, headings, and body of the text. For example, for a document containing the theme of "machine learning," it extracts keywords such as "neural network" and "regression analysis." It then evaluates the importance of the extracted keywords and themes, as well as each section, and stores this information as metadata in the document database.

[0410] Step 3: Emotion recognition by the emotion engine

[0411] Input: User's voice or facial image data

[0412] Output: User's emotional information

[0413] Specific behavior:

[0414] Device: When a user searches, the device collects the user's voice data or facial image data. This is done using a microphone or camera, which records and photographs the data in real time.

[0415] Server: The emotion engine analyzes the collected voice data and facial image data to recognize the user's emotional state. For voice data, it uses the emotion recognition API from Microsoft Azure Cognitive Services, and for facial image data, it uses OpenCV and dlib algorithms to identify whether the user is anxious or calm. This emotional information is stored in a database.

[0416] Step 4: Entering and Parsing Search Prompts

[0417] Input: The search prompt entered by the user

[0418] Output: The generated search query

[0419] Specific behavior:

[0420] User: The user enters a keyword or phrase related to the information they are looking for as a prompt in the search screen. For example, they might enter "latest applications of machine learning."

[0421] Terminal: Sends the entered prompt to the server. The prompt is encoded as string data and sent through an HTTP request.

[0422] Server: The server analyzes the received prompt using a natural language processing model (e.g., GPT-3) to extract relevant themes and keywords, such as "machine learning," "latest," and "applied," and generates a search query based on these.

[0423] Step 5: Search and suggest related materials

[0424] Input: Generated search query and sentiment information

[0425] Output: Related materials picked up

[0426] Specific behavior:

[0427] Server: Using the generated search query, the server searches the resource database. It uses information retrieval algorithms such as TF-IDF and Doc2Vec to match the resource database metadata and find relevant resources.

[0428] Server: Extracts important slides and sections from each presentation. It also customizes the order and content of presentation materials based on the user's perceived emotions. For example, if the user is feeling impatient, it prioritizes short, focused presentation materials.

[0429] Step 6: View the results

[0430] Input: Customized proposal list

[0431] Output: Display the list of materials to the user

[0432] Specific behavior:

[0433] Terminal: The terminal displays the customized list of proposal materials sent from the server to the user. For each material, the title, summary, keywords, and thumbnails of important slides are displayed in a user interface using HTML and CSS.

[0434] User: The user can view the list of proposal documents and find the ones they are interested in for detailed view or download. For example, they can click on the title of a proposal document to access its detailed page and download it in PDF format.

[0435] The above is the flow of program processing for this system.

[0436] (Application example 2)

[0437] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0438] In today's information-saturated society, it is difficult for users to quickly and accurately obtain the specific information they are looking for. In particular, there is no method to efficiently search through materials stored in electronic files and provide appropriate information based on the user's emotions and situation. Furthermore, online shopping sites are in need of a system that can perform personalized product searches that take the user's emotions into account.

[0439] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0440] In this invention, the server includes means for receiving electronic files of reference materials, means for converting the contents of the received electronic files into text data using optical character recognition technology, means for analyzing the text data using a natural language processing engine to extract keywords and themes and generate metadata, and means for recognizing the user's emotions and customizing search results based on the user's emotional state. This enables the user to quickly and accurately obtain the information they are looking for, and further enables the provision of personalized information according to the user's emotions.

[0441] "Electronic reference files" are digital files that store information uploaded by users to the system, including formats such as PDF and PPT.

[0442] "Optical character recognition technology" is a technology that extracts text data from image data, and is also known as OCR (Optical Character Recognition).

[0443] "Text data" refers to digital data that contains textual information and has been converted into a format that can be used for analysis and searching.

[0444] A "natural language processing engine" is a computer program for understanding and analyzing human language, and is used to extract keywords and themes from text data.

[0445] "Metadata" is data that contains information about data, such as keywords, themes, and importance of text data stored in a material database.

[0446] A "database" is a collection of data that is stored and managed electronically and that can be searched and retrieved by users.

[0447] A "search prompt" is a keyword or phrase that a user uses to input the information they are seeking and is used in the system's search function.

[0448] "Emotional state" refers to the user's current emotional state, which the system recognizes by analyzing voice data and image data.

[0449] "Personalization" refers to adapting and customizing information based on a user's individual characteristics and feelings.

[0450] "Important sections" refer to parts or slides that are considered to be particularly important in the materials, and are extracted to quickly provide users with the information they need.

[0451] This invention is a system that recognizes a user's emotions and customizes search results based on a specific emotional state. This system is particularly applicable to online shopping sites, where users not only input images and keywords of the items they want, but also recognize emotions from facial images and voice, and suggest products that best suit those emotions. This system is primarily composed of three entities: a server, a terminal, and the user.

[0452] Server configuration and roles

[0453] The server has the following functions:

[0454] 1. Receiving and processing electronic files: Electronic files uploaded by users are received and converted into text data using OCR technology. This text data is then analyzed using a natural language processing engine to extract keywords and themes, generating metadata, and storing it in the material database.

[0455] 2. Emotion recognition: It has the function of recognizing the user's emotions, and analyzes the voice data and facial image data collected by the emotion engine to identify the user's emotional state.

[0456] 3. Search prompt analysis and customization: Analyze the search prompt received from the user and extract relevant keywords, then customize the search results based on the user's emotional state.

[0457] 4. Product Search and Recommendation: The generated search query is used to search the document database to find relevant products, and then personalized product recommendations are made based on the user's emotional state.

[0458] Device configuration and role

[0459] The terminal has the following functions:

[0460] 1. Uploading an electronic file: The user accesses the upload screen of the system, selects the electronic file they want to use as reference, and clicks the upload button.

[0461] 2. Entering a search prompt: The user enters an image or keyword about the material or product they want as a prompt.

[0462] 3. Emotion data collection: The emotion engine collects the user's voice data and facial image data and sends them to the server.

[0463] 4. Display search results: Display a customized list of suggested products and key information about each product to the user.

[0464] User operations

[0465] The user uses the system in the following steps:

[0466] 1. Uploading an electronic file: The user selects the electronic file they want to use as reference from the upload screen on their device and clicks the upload button. The device then sends the file to the server.

[0467] 2. Enter a search prompt: Enter an image or keyword of the material or product you want as a prompt. For example, enter "latest smartphone."

[0468] 3. Emotion recognition: The camera captures the user's facial image and voice, and the emotion engine analyzes the data to identify the user's emotional state. For example, it recognizes the emotion of "anxiety."

[0469] 4. Review search results: Review the customized list of suggested products sent from the server, and proceed to the detailed view or purchase process for the products that interest you.

[0470] Specific examples

[0471] For example, if a user types in "latest smartphone" and the emotion "impatient" is recognized from an image captured by the camera, the generated search prompt might look like this:

[0472] Prompt statement:

[0473] The latest smartphones with a feeling of anxiety

[0474] Based on this prompt, the server will provide quick and focused results, such as simpler, more readily available models and items that are in stock and ready to buy.

[0475] In this way, the system provides optimal products according to the user's emotional state and enables the user to quickly and efficiently obtain the information they are looking for.

[0476] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0477] Step 1:

[0478] The user accesses the system's upload screen, selects the electronic file they want to use as reference, and clicks the upload button. The terminal then sends this electronic file to the server. The input is the electronic file selected by the user, and the output is the transfer of the electronic file to the server.

[0479] Step 2:

[0480] The server stores the received electronic files and converts the file contents into text data using optical character recognition (OCR). The input is an electronic file, and the output is text data. This text data is organized into sections that separate each slide or page, and stored in a materials database.

[0481] Step 3:

[0482] The server analyzes the text data stored in the resource database using a natural language processing (NLP) engine, extracting key keywords and themes from titles, headings, and main text content. The input is text data, and the output is metadata (extracted keywords, themes, and importance information). This metadata is stored in the resource database.

[0483] Step 4:

[0484] The user enters a search prompt, and the terminal sends the prompt to the server. The input is the prompt sentence entered by the user, and the output is the transfer of the prompt sentence to the server.

[0485] Step 5:

[0486] The server analyzes the received search prompt and extracts relevant keywords. The input is the search prompt sentence and the output is the extracted keywords. The server then generates a search query based on these keywords.

[0487] Step 6:

[0488] The terminal transmits the user's voice data and facial image data to the emotion engine, which analyzes the data and recognizes the user's emotional state. The input is the voice data and facial image data, and the output is the recognized emotional state.

[0489] Step 7:

[0490] The server uses the generated search query to search a database of materials and pick out highly relevant materials and products. It also customizes search results based on the user's emotional state. For example, for a user who is feeling impatient, it prioritizes displaying short, focused materials and products. The input is the search query and the user's emotional state, and the output is customized search results.

[0491] Step 8:

[0492] The server sends customized proposal materials and product lists to the terminal. The terminal displays this information to the user. The input is the customized search results sent from the server, and the output is the display of the search results to the user. The user can view details of the materials or products they are interested in, download them, or proceed to purchase them.

[0493] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0494] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0495] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0496] [Second embodiment]

[0497] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0498] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0499] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0500] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0501] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0502] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0503] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0504] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0505] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0506] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0507] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0508] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0509] The present invention is a system for efficiently searching and suggesting reference materials, particularly those stored as electronic files, and automatically processing them to enable users to quickly and accurately obtain the information they desire.

[0510] This system is mainly composed of three entities: a server, a terminal, and a user. The specific operation of each is explained below.

[0511] Uploading and storing materials

[0512] User: First, the user accesses the system's upload screen, selects the electronic file (e.g., PDF, PPT, etc.) they want to use as reference, and clicks the upload button.

[0513] Terminal: Sends the file selected by the user to the server.

[0514] Server: The received file is stored in temporary storage. Then, using OCR (Optical Character Recognition) technology, the contents of the uploaded file are converted into text data. After conversion, each slide or page in the file is divided into sections and assigned a unique ID. This text data and section information are stored in a material database.

[0515] Text analysis and feature extraction of materials

[0516] Server: Analyzes text data stored in the resource database using a natural language processing (NLP) engine. Extracts key keywords and themes from titles, headings, and main text, and evaluates the importance of each section. The extracted keywords, themes, and importance information are stored in the resource database as metadata.

[0517] Entering and parsing search prompts

[0518] User: Enter the image or keywords of the desired material as a prompt.

[0519] Terminal: Sends the entered prompt to the server.

[0520] Server: Analyzes the received prompts, extracts relevant themes and keywords, and generates search queries based on the extracted themes and keywords.

[0521] Search and suggest related materials

[0522] Server: Searches the resource database using the generated search query. It compares the metadata in the resource database to select multiple highly relevant resources. It then extracts important slides and sections from each of the selected resources and generates summary information for each (title, keywords, importance, etc.). It then sends a list of the selected resources and key sections to the user's device.

[0523] Displaying the results

[0524] Terminal: Displays the user a list of proposal materials and the important slides and sections of each material. For each material, the title, summary, keywords, and thumbnails of important slides are displayed.

[0525] Users: Review the proposed materials and, if interested, view details or download them.

[0526] Specific examples

[0527] For example, consider the case where a user uploads a slide set for a conference presentation, "Research Presentation_Machine Learning.ppt." When the user selects a file using the upload screen and clicks the upload button, the device sends the file to the server. The server saves the received file in temporary storage and converts it into text data using OCR technology. The converted text data is divided into sections, and the information is stored in a materials database.

[0528] Next, when a user enters a prompt such as "latest applications of machine learning" into the search screen, the prompt is sent from the device to the server. The server analyzes the prompt and extracts keywords such as "machine learning," "latest," and "application." A search query is generated based on the extracted keywords, and by searching the document database, highly relevant documents are picked up.

[0529] Important slides (e.g., slides 3 and 7) are extracted from the selected materials, and summary information is generated for each. This information is sent to the user's device and displayed as a list of proposed materials.

[0530] Users can select the materials they want to know more about from the suggested list and view or download them, allowing them to obtain the information they are looking for quickly and efficiently.

[0531] Thus, the present invention is significantly more efficient than existing document retrieval systems and can quickly provide information that meets the user's needs.

[0532] The processing flow will be explained below.

[0533] Step 1: Upload your materials

[0534] User: The user accesses the system's upload screen, selects the electronic file (e.g., PDF, PPT, etc.) they want to use as reference, and clicks the upload button.

[0535] Step 2: Send the file

[0536] Terminal: Sends the file selected by the user to the server.

[0537] Step 3: Save the file

[0538] Server: Stores received files in temporary storage.

[0539] Step 4: OCR processing

[0540] Server: Launches an OCR (optical character recognition) engine and converts the contents of the uploaded file into text data.

[0541] Step 5: Sectioning

[0542] Server: Each slide or page in the text data file is divided into sections, and a unique ID is assigned to each section.

[0543] Step 6: Store in the database

[0544] Server: Stores the converted text data and section information in the material database.

[0545] Step 7: Natural Language Processing Analysis

[0546] Server: Analyzes the text data stored in the material database using a natural language processing (NLP) engine.

[0547] Step 8: Extracting keywords and themes

[0548] Server: Extracts key keywords and themes from titles, headings, and body content, and rates the importance of each section.

[0549] Step 9: Generate Metadata

[0550] Server: Generates extracted keywords, themes, and importance information as metadata and adds them to the material database.

[0551] Step 10: Enter the search prompt

[0552] User: Accesses the system's search screen and enters the image or keywords of the desired material as prompted.

[0553] Step 11: Sending prompts

[0554] Terminal: Sends the entered prompt to the server.

[0555] Step 12: Parse prompts

[0556] Server: Parses the received prompts and extracts relevant themes and keywords.

[0557] Step 13: Generate a search query

[0558] Server: Generates search queries based on the extracted themes and keywords.

[0559] Step 14: Search the documentation database

[0560] Server: Searches the resource database using the generated search query.

[0561] Step 15: Identify relevant materials

[0562] Server: Compares the metadata in the material database and selects multiple highly relevant materials.

[0563] Step 16: Extracting important sections

[0564] Server: Extracts key slides and sections from each document.

[0565] Step 17: Generate summary information

[0566] Server: Generates summary information (title, keywords, importance, etc.) for the extracted important slides and sections.

[0567] Step 18: Submit your proposal list

[0568] Server: Sends the list of proposal materials and the list of important sections to the user's terminal.

[0569] Step 19: View the results

[0570] Terminal: Displays the user a list of proposal materials and key slides and sections of each material.

[0571] Step 20: View details and download

[0572] Users: Review the suggested materials and select the ones that interest them for detailed viewing or download.

[0573] In this way, the system provides the user with the materials they require efficiently and quickly through a series of steps.

[0574] Example 1

[0575] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0576] Conventional reference material search systems have limitations in their ability to efficiently search and suggest the contents of electronic files, making it difficult for users to quickly and accurately obtain the information they need. Furthermore, they lacked the technology to improve the accuracy of suggesting highly relevant materials and the ability to extract and display important sections of materials. This resulted in the problem of users spending a great deal of time and effort finding the information they were looking for.

[0577] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0578] In this invention, the server includes: means for receiving electronic files of reference materials; means for converting the contents of the received electronic files into text data using optical character recognition technology; means for analyzing the text data with a natural language processing engine to extract keywords and themes and generate metadata; means for storing the metadata in a database; means for receiving search prompts and extracting relevant keywords from the prompts; means for searching the database based on the extracted keywords to select relevant materials; means for extracting important sections from the selected materials and providing them to the user; means for assigning unique IDs to the extracted sections and evaluating their importance; means for extracting themes from the prompts using a machine learning model; means for recommending highly relevant materials based on the metadata of the materials stored in the database; means for generating summary information of the relevant materials and transmitting it to the user terminal; and means for analyzing the search prompts and generating search queries. This allows users to quickly and efficiently obtain the information they need and effectively search for and suggest highly relevant materials.

[0579] "Electronic reference files" refers to materials and documents stored in digital formats such as PDF and PPT.

[0580] The "means for receiving" refers to a function for acquiring an electronic file when the server receives an uploaded file from a user.

[0581] "Optical character recognition technology" is a technology called OCR (Optical Character Recognition), which analyzes characters in an image and converts them into text data.

[0582] "Text data" refers to data in which only character information is extracted from the contents of an electronic file and saved in text format.

[0583] A "natural language processing engine" is a computer science technology for analyzing text data to extract keywords and themes.

[0584] "Metadata" refers to data that includes keywords, themes, section importance information, and the like extracted from text data.

[0585] A "database" is a system for efficiently storing, retrieving, and managing various types of data.

[0586] A "search prompt" is a keyword or phrase that a user enters to search for the information they need.

[0587] "Means for extracting relevant keywords" refers to a function for extracting important keywords from a search prompt.

[0588] A "search query" is a specific search condition generated to search a database.

[0589] "Means for picking out relevant materials" refers to a function for selecting relevant materials from a database based on a search query.

[0590] "Means for extracting important sections" refers to a function for selecting particularly important parts from the picked-up material.

[0591] A "unique ID" is an identifier assigned to each section to uniquely identify it.

[0592] The "means for evaluating the importance" refers to a function for quantitatively evaluating the importance of the extracted section.

[0593] A "machine learning model" is an algorithm or technique that learns from data and makes predictions or classifications based on that data.

[0594] "Means for recommending highly relevant materials" refers to a function for selecting and suggesting materials that best fit the user's search prompt.

[0595] The "means for generating summary information and transmitting it to the user terminal" refers to a function for generating a summary of the picked-up material and transmitting it to the user terminal.

[0596] The present invention is a system for efficiently searching and suggesting reference materials. This system automatically processes materials stored as electronic files, enabling users to quickly and accurately obtain the information they desire. The system is primarily composed of three entities: a server, a terminal, and a user. The specific operation of each is described below.

[0597] Uploading and storing materials

[0598] Users access the system's upload screen, select the electronic file they want to use as reference (e.g. PDF, PPT, etc.), and click the upload button. The terminal sends the file selected by the user to the server. The server stores the received file in temporary storage. The server then uses OCR (optical character recognition) technology to convert the contents of the uploaded file into text data. Tesseract OCR is used for the conversion. After conversion, each slide or page in the file is divided into sections and assigned a unique ID. This text data and section information are stored in a material database. The material database uses MySQL.

[0599] Text analysis and feature extraction of materials

[0600] The server analyzes the text data stored in the document database using a natural language processing (NLP) engine. The NLP engine used is SpaCy. It extracts key keywords and themes from titles, headings, and body content, and uses the TF-IDF algorithm to evaluate the importance of each section. The extracted keywords, themes, and importance information are stored in the document database as metadata.

[0601] Entering and parsing search prompts

[0602] The user enters an image of the desired material or keywords as a prompt into the search screen. The device sends the entered prompt to the server. The server analyzes the received prompt and extracts related themes and keywords. The BERT model is used to analyze the prompt. A search query is generated based on the extracted keywords.

[0603] Search and suggest related materials

[0604] The server searches the document database using the generated search query. It compares the results with the metadata in the document database to select multiple highly relevant documents. It calculates a relevance score and selects the top documents. It extracts the most important slides and sections from each of the selected documents and generates summary information (title, keywords, importance, etc.) for each. The generated summary information is sent from the server to the user's device in JSON format.

[0605] Displaying the results

[0606] The terminal displays a list of proposed materials and the important slides and sections of each material to the user. For each material, the title, summary, keywords, and thumbnails of important slides are displayed. HTML / CSS and JavaScript are used to display the user interface. The user can check the proposed materials and, if interested, can view details or download them. Downloads are performed using an HTTP GET request.

[0607] Specific examples

[0608] For example, consider the case where a user uploads a slide set for a conference presentation, "Research Presentation_Machine Learning.ppt." When the user selects a file using the upload screen and clicks the upload button, the device sends the file to the server. The server saves the received file in temporary storage and converts it into text data using OCR technology. The converted text data is divided into sections, and the information is stored in a materials database.

[0609] Next, if a user enters a prompt such as "latest applications of machine learning" into the search screen, the prompt is sent from the device to the server. The server analyzes the prompt and extracts keywords such as "machine learning," "latest," and "application." A search query is generated based on the extracted keywords, and highly relevant materials are picked out by searching the materials database. Important slides (e.g., slides 3 and 7) are extracted from the picked materials, and summary information is generated for each. This information is sent to the user's device and displayed as a list of suggested materials.

[0610] This system allows users to quickly and efficiently obtain the information they need, significantly improving the efficiency of research and work. As a result, it becomes possible to provide information that meets the user's needs appropriately.

[0611] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0612] Step 1:

[0613] The user accesses the upload screen of the system, selects the electronic file they want to use as reference, and clicks the upload button.

[0614] Input: An electronic file selected by the user (e.g. PDF, PPT, etc.)

[0615] Output: Request to upload selected files

[0616] Step 2:

[0617] The terminal transmits the file selected by the user to the server.

[0618] Input: User upload request and selected electronic file

[0619] Output: A request for an electronic file to be sent to the server

[0620] Step 3:

[0621] The server stores the received file in temporary storage.

[0622] Input: Electronic file sent from the terminal

[0623] Output: Electronic file saved in temporary storage

[0624] Step 4:

[0625] The server uses OCR technology to convert the contents of the uploaded file into text data.

[0626] Input: Electronic files stored in temporary storage

[0627] Output: Converted text data

[0628] Step 5:

[0629] The server divides the text data into sections and assigns each a unique ID.

[0630] Input: Converted text data

[0631] Output: Text data with a unique ID assigned to each section

[0632] Step 6:

[0633] The server stores the section information and text data in a material database.

[0634] Input: Text data divided into sections

[0635] Output: Text data stored in the material database and its section information

[0636] Step 7:

[0637] The server uses a natural language processing engine to analyze the text data stored in the document database, extract key keywords and themes, and evaluate the importance of each section.

[0638] Input: Text data stored in the material database

[0639] Output: Extracted keywords, themes, and importance of each section

[0640] Step 8:

[0641] The server stores the extracted keywords, themes, and importance information as metadata in the document database.

[0642] Input: Extracted keywords, themes, and importance information

[0643] Output: Metadata stored in the material database

[0644] Step 9:

[0645] The user enters an image of the material they want or keywords into the search screen as a prompt.

[0646] Input: Search prompt

[0647] Output: User search request

[0648] Step 10:

[0649] The terminal sends the entered prompt to the server.

[0650] Input: User search request

[0651] Output: Sending a search prompt to the server

[0652] Step 11:

[0653] The server analyzes the prompts it receives and uses machine learning models to extract relevant themes and keywords.

[0654] Input: Submitted search prompt

[0655] Output: Themes and keywords extracted from the prompt

[0656] Step 12:

[0657] The server generates a search query based on the extracted themes and keywords.

[0658] Input: Extracted themes and keywords

[0659] Output: Generated search query

[0660] Step 13:

[0661] The server searches the document database using the generated search query and picks out multiple highly relevant documents.

[0662] Input: Generated search query

[0663] Output: Related materials picked up

[0664] Step 14:

[0665] The server extracts the most important slides and sections from each document and generates a summary.

[0666] Input: Picked up materials

[0667] Output: Summary information (title, keywords, importance, etc.)

[0668] Step 15:

[0669] The server transmits a list of proposal materials including summary information to the user's terminal.

[0670] Input: Summary Information

[0671] Output: A list of proposal materials sent to the user's device

[0672] Step 16:

[0673] The device displays a list of proposal materials and important slides and sections of each material to the user.

[0674] Input: Proposal list sent from the server

[0675] Output: A list of proposals displayed to the user

[0676] Step 17:

[0677] The user can review the proposed materials and, if interested, view them in detail or download them.

[0678] Input: Displayed proposal list

[0679] Output: View detailed view or download document

[0680] Through the above steps, the user can quickly and efficiently obtain the information he or she needs.

[0681] (Application example 1)

[0682] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0683] Factories and industrial sites often contain vast amounts of technical documents and manuals, but it is not easy to quickly and accurately search through them and obtain the information needed. Managing and searching through these documents takes a great deal of time and effort, especially when there are many paper documents and printed materials. This situation reduces work efficiency and impacts productivity. Furthermore, delays in accessing necessary information can lead to delays in equipment maintenance and troubleshooting. To address these challenges, an effective document management and search system is needed.

[0684] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0685] In this invention, the server includes: means for receiving electronic files of reference materials; means for converting the contents of the received electronic files into text data using optical character recognition technology; means for analyzing the text data with a natural language processing engine to extract keywords and themes and generate metadata; means for storing a database for storing the metadata; means for receiving search prompts and extracting relevant keywords from the prompts; means for searching a database based on the extracted keywords to find relevant materials; means for extracting important sections from the found materials and providing them to a user; means for a user to scan paper documents or printed materials with a smartphone and upload the images; means for processing the uploaded images with OCR technology and converting them to text; and means for searching a database using a TF-IDF-based search algorithm to suggest relevant materials in response to a user's search query, enabling factories and industrial sites to quickly and accurately search for necessary materials and provide them to users.

[0686] "Electronic files of reference materials" are technical documents and manuals related to work in factories and industrial sites that are stored in digital format.

[0687] Optical character recognition (OCR) is a technology that scans characters written on paper documents or printed materials and converts them into digital character data.

[0688] "Text data" refers to the text information in electronic files converted using OCR technology and stored in digital format.

[0689] A "natural language processing engine (NLP)" is a program that analyzes text data and automatically extracts important keywords and themes.

[0690] "Metadata" refers to additional information extracted from text data, such as keywords, themes, and importance, which is used to efficiently search and organize data within a database.

[0691] A "database" is a system for centrally managing and storing metadata and related material information.

[0692] A "search prompt" is a keyword or query that a user enters to specifically describe the information they are seeking.

[0693] A "TF-IDF-based search algorithm" is a statistical method for evaluating the importance of documents in a database based on a search query and deriving highly relevant materials.

[0694] "Scanning with a smartphone" refers to the process by which a user uses the smartphone's camera function to convert paper documents or printed materials into digital images.

[0695] "Uploading" is the act of a user transferring a digital image or electronic file to a cloud system such as a server.

[0696] "Important sections" are those parts of the selected material that are particularly relevant and useful to the user.

[0697] The present invention is a document management and search system for use in factories and industrial sites. This system is composed of three main components: a server, a terminal, and a user. The specific operation of each component will be explained below.

[0698] Uploading and storing materials

[0699] User: The user uses their smartphone to scan paper documents or printed materials and convert them into digital images, then selects the scanned image on the application's upload screen and clicks the upload button.

[0700] Terminal: Sends the image file selected by the user to the server.

[0701] Server: The received image files are stored in temporary storage and the contents are converted into text data using OCR (Optical Character Recognition) technology. After conversion, each file is divided into sections and assigned a unique ID. This text data and section information are stored in a material database.

[0702] Text analysis and feature extraction of materials

[0703] Server: Analyzes text data stored in the resource database using a natural language processing (NLP) engine. Extracts key keywords and themes and evaluates the importance of each section. The extracted keywords, themes, and importance information are stored in the resource database as metadata.

[0704] Entering and parsing search prompts

[0705] User: The user enters a search prompt into the application for the required material, for example, "Basic steps in PLC programming."

[0706] Terminal: Sends the entered prompt to the server.

[0707] Server: Analyzes the received prompts, extracts relevant themes and keywords, and generates search queries based on the extracted themes and keywords.

[0708] Search and suggest related materials

[0709] Server: Searches the document database using the generated search query. It uses a TF-IDF-based search algorithm to evaluate the importance of documents in the database and selects highly relevant documents. It then extracts important slides and sections from each of the selected documents and generates summary information for each (title, keywords, importance, etc.). It then sends a list of the selected documents and key sections to the user's device.

[0710] Displaying the results

[0711] Terminal: Displays the user a list of proposed materials and the important sections of each material. For each material, displays the title, summary, keywords, and thumbnails of important sections.

[0712] Users: Review the proposed materials and, if interested, view details or download them.

[0713] The system's hardware mainly requires a smartphone with a camera and a high-performance server. It uses the pytesseract library for OCR technology, and the nltk and scikit-learn libraries for natural language processing. The TF-IDF algorithm is used for data analysis of the documents.

[0714] Specific examples

[0715] For example, consider a user on a factory floor looking for a "Troubleshooting Guide for PLC Programming." The user scans a paper manual with their smartphone and uploads it to the server using an application. Next, they input "Basic Steps for PLC Programming" as a search prompt. The server analyzes this prompt and uses a TF-IDF-based algorithm to search and suggest relevant documents from the database. The user then checks the detailed view of the displayed document list to obtain the information they need.

[0716] The above method makes it possible to quickly and accurately search for and provide users with the necessary documents and information at factories and industrial sites. This system not only improves work efficiency at the site, but also speeds up troubleshooting and maintenance work by providing timely and appropriate information.

[0717] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0718] Step 1:

[0719] A user uses a smartphone to scan paper documents or printed materials and convert them into digital images. Then, they select the scanned image on the application's upload screen and click the upload button. At this point, the input is the scanned image file, and the output is an upload request.

[0720] Step 2:

[0721] The terminal sends the image file selected by the user to the server. The input here is the scanned image file, and the output is the image file sent to the server. This sending process is performed via the Internet.

[0722] Step 3:

[0723] The server stores the received image file in temporary storage. It then uses OCR technology to convert the image file into text data. The input here is the sent image file, and the output is text data. Specifically, it uses the pytesseract library to recognize the text in the image and convert it into text.

[0724] Step 4:

[0725] The server divides the converted text data into sections, assigns unique IDs, and stores them in a database. In this step, the input is text data, and the output is text data in a database divided into sections.

[0726] Step 5:

[0727] The server uses a natural language processing engine (NLP) to analyze the text data stored in the resource database and extract key keywords and themes. The input is the text data in the database, and the output is metadata. Specifically, it uses NLP technology to extract important words and phrases from the text and stores them as metadata.

[0728] Step 6:

[0729] The user enters a search prompt for the required material on the application search screen, for example, a keyword such as "Basic steps of PLC programming." The input of this step is the search prompt, and the output is a prompt transmission request from the terminal to the server.

[0730] Step 7:

[0731] The terminal sends the entered prompt to the server, where the input is the search prompt entered by the user and the output is a prompt transmission request to the server.

[0732] Step 8:

[0733] The server analyzes the prompts it receives, extracts relevant themes and keywords, and generates a search query based on that information. The input is the prompt, and the output is the generated search query. This analysis is performed using a generative AI model.

[0734] Step 9:

[0735] The server uses the generated search query to search the resource database, where the input is the search query and the output is a list of relevant resources. A TF-IDF-based search algorithm is used to evaluate and select relevant documents.

[0736] Step 10:

[0737] The server extracts important sections from each of the documents picked up and generates summary information (title, keywords, importance, etc.) for each. The input is a list of documents, and the output is a list of proposed documents with summary information.

[0738] Step 11:

[0739] The server sends the list of proposal materials and information on important sections of each material to the user's terminal. The input is the list of proposal materials with summary information, and the output is a transmission request to the user's terminal.

[0740] Step 12:

[0741] The terminal displays the list of proposal materials and important sections of each material to the user. The input is the list of proposal materials sent from the server, and the output is the material information displayed to the user.

[0742] Step 13:

[0743] The user checks the proposed materials and, if interested, can view details or download them. The input is the material information displayed to the user, and the output is the downloaded detailed materials.

[0744] This series of processes makes it possible to quickly and accurately search for necessary materials and information at factories and industrial sites and provide them to users.

[0745] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0746] The present invention is a system for efficiently searching electronic files of reference materials and quickly providing users with the information they desire. Furthermore, the present invention has the ability to recognize a user's emotions and customize search results based on those emotions. This system is specifically designed to automatically process materials stored as electronic files, enabling users to quickly and accurately obtain the information they desire.

[0747] This system is mainly composed of three entities: a server, a terminal, and a user. The specific operation of each is explained below.

[0748] Uploading and storing materials

[0749] User: The user accesses the system's upload screen, selects the electronic file (e.g., PDF, PPT, etc.) they want to use as reference, and clicks the upload button.

[0750] Terminal: Sends the file selected by the user to the server.

[0751] Server: The received file is stored in temporary storage. Then, using OCR (Optical Character Recognition) technology, the contents of the uploaded file are converted into text data. After conversion, each slide or page in the file is divided into sections and assigned a unique ID. This text data and section information are stored in a material database.

[0752] Text analysis and feature extraction of materials

[0753] Server: Analyzes text data stored in the resource database using a natural language processing (NLP) engine. Extracts key keywords and themes from titles, headings, and main text, and evaluates the importance of each section. The extracted keywords, themes, and importance information are stored in the resource database as metadata.

[0754] Emotion recognition by emotion engine

[0755] On the device: When a user searches, the emotion engine collects the user's voice data or facial image data.

[0756] Server: The emotion engine analyzes the collected data and recognizes the user's emotional state, for example, identifying whether the user is anxious or calm.

[0757] Entering and parsing search prompts

[0758] User: Enter the image or keywords of the desired material as a prompt.

[0759] Terminal: Sends the entered prompt to the server.

[0760] Server: Analyzes the received prompts, extracts relevant themes and keywords, and generates search queries based on the extracted themes and keywords.

[0761] Search and suggest related materials

[0762] Server: Searches the resource database using the generated search query, compares it with the resource database metadata, and selects multiple highly relevant resources.

[0763] Server: Extracts important slides and sections from each presentation and customizes the order and content of the presentation based on the user's perceived emotions. For example, prioritize short, concise presentations for impatient users.

[0764] Displaying the results

[0765] Terminal: Presents users with a customized list of presentation materials and key slides and sections for each material. For each presentation, it displays the title, summary, keywords, and thumbnails of key slides.

[0766] Users: Review the proposed materials and, if interested, view details or download them.

[0767] Specific examples

[0768] For example, consider the case where a user uploads a slide set for a conference presentation, "Research Presentation_Machine Learning.ppt." When the user selects a file using the upload screen and clicks the upload button, the device sends the file to the server. The server saves the received file in temporary storage and converts it into text data using OCR technology. The converted text data is divided into sections, and the information is stored in a materials database.

[0769] Next, when a user enters a prompt such as "latest applications of machine learning" into the search screen, the prompt is sent from the device to the server. The server analyzes the prompt and extracts keywords such as "machine learning," "latest," and "application." A search query is generated based on the extracted keywords, and by searching the document database, highly relevant documents are picked up.

[0770] At that time, the device sends the user's voice data and facial image data to the emotion engine, which then recognizes the user's emotions.The server then customizes the presentation order and content of the selected materials based on that emotional information.For example, if the user is feeling impatient, it will prioritize displaying materials that summarize only the main points.

[0771] Finally, the list of suggested materials is sent to the terminal, where the user can review the list and view or download the materials they are interested in. In this way, the system provides the most appropriate materials according to the user's emotional state, enabling the user to quickly and efficiently obtain the information they are looking for.

[0772] The processing flow will be explained below.

[0773] Step 1: Upload your materials

[0774] User: The user accesses the system's upload screen, selects the electronic file (e.g., PDF, PPT, etc.) they want to use as reference, and clicks the upload button.

[0775] Step 2: Send the file

[0776] Terminal: Sends the file selected by the user to the server.

[0777] Step 3: Save the file

[0778] Server: Stores received files in temporary storage.

[0779] Step 4: OCR processing

[0780] Server: Launches an OCR (optical character recognition) engine and converts the contents of the uploaded file into text data.

[0781] Step 5: Sectioning

[0782] Server: Each slide or page in the text data file is divided into sections, and a unique ID is assigned to each section.

[0783] Step 6: Store in the database

[0784] Server: Stores the converted text data and section information in the material database.

[0785] Step 7: Natural Language Processing Analysis

[0786] Server: Analyzes the text data stored in the material database using a natural language processing (NLP) engine.

[0787] Step 8: Extracting keywords and themes

[0788] Server: Extracts key keywords and themes from titles, headings, and body content, and rates the importance of each section.

[0789] Step 9: Generate Metadata

[0790] Server: Generates extracted keywords, themes, and importance information as metadata and adds them to the material database.

[0791] Step 10: Enter the search prompt

[0792] User: Accesses the system's search screen and enters the image or keywords of the desired material as prompted.

[0793] Step 11: Sending prompts

[0794] Terminal: Sends the entered prompt to the server.

[0795] Step 12: Parse prompts

[0796] Server: Parses the received prompts and extracts relevant themes and keywords.

[0797] Step 13: Generate a search query

[0798] Server: Generates search queries based on the extracted themes and keywords.

[0799] Step 14: Collect emotion data

[0800] Device: In parallel with entering the search prompt, the user's voice data and facial image data are sent to the emotion engine.

[0801] Step 15: Analyze the emotion data

[0802] Server: The emotion engine analyzes the collected voice data and facial image data to recognize the user's emotional state, for example, determining whether they are "anxious" or "calm."

[0803] Step 16: Search the documentation database

[0804] Server: Searches the resource database using the generated search query.

[0805] Step 17: Identify relevant materials

[0806] Server: Compares the metadata in the material database and selects multiple highly relevant materials.

[0807] Step 18: Extracting important sections

[0808] Server: Extracts key slides and sections from each document.

[0809] Step 19: Emotional Customization

[0810] Server: Customize the presentation order and content of the selected materials based on the user's recognized emotional state. For example, prioritize displaying materials that summarize the main points to a user who is feeling impatient.

[0811] Step 20: Submit your proposal list

[0812] Server: Sends a customized list of proposal materials and a list of important sections to the user's device.

[0813] Step 21: View the results

[0814] Terminal: Displays the user a list of proposal materials and key slides and sections of each material.

[0815] Step 22: View details and download

[0816] Users: Review the suggested materials and select those that interest them for detailed viewing or download.

[0817] Example 2

[0818] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0819] Conventional document search systems have difficulty in quickly and accurately providing the information users are looking for. Furthermore, because they provide uniform search results without considering the user's emotions or state, it is difficult to provide information that meets the user's needs. Furthermore, there are often too many search results, which can lead to important information being overlooked, resulting in a decrease in user satisfaction.

[0820] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0821] In this invention, the server includes means for receiving electronic files of reference materials, means for converting the contents of the received electronic files into text data using optical character recognition technology, means for analyzing the text data with a natural language processing engine to extract keywords and themes and generate metadata, means for analyzing audio data or image data and recognizing the user's emotions, means for searching a database based on the extracted keywords and the recognized emotions to select relevant materials, and means for extracting important sections from the selected materials and providing them to the user. This makes it possible to provide appropriate information according to the user's emotional state, enabling the user to quickly and accurately obtain the information they are looking for.

[0822] "Reference materials" refers to electronic files that users use to obtain information, such as PDFs and PPTs.

[0823] "Optical character recognition technology" is a technology that reads characters from image data and converts them into text data.

[0824] "Text data" refers to data that contains only text information and that has been converted using optical character recognition technology.

[0825] "Natural language processing engine" is a general term for software and algorithms that analyze text data and extract keywords and themes.

[0826] "Metadata" refers to data attribute information, such as keywords, themes, and relevance evaluation information extracted from text data.

[0827] A "search prompt" refers to the keywords or phrases that a user enters about the search target.

[0828] "Emotion recognition" is a technology that analyzes voice data and image data to identify a user's emotional state.

[0829] "Database" refers to an information system that stores data in a structured way to facilitate retrieval and access.

[0830] "Key sections" are those parts of the document that are assessed as being particularly important and include key points and summary information.

[0831] A "thumbnail" is a small version of a document or image that allows you to easily check the content.

[0832] A "machine learning model" refers to an algorithm or model that learns patterns based on data and makes predictions and classifications.

[0833] The present invention is a system for efficiently searching electronic files of reference materials and quickly providing users with the information they desire. Furthermore, the present invention has the ability to recognize users' emotions and customize search results based on those emotions. This system is specifically designed to automatically process materials stored as electronic files, enabling users to quickly and accurately obtain the information they desire.

[0834] This system is mainly composed of three entities: a server, a terminal, and a user. The specific operations performed by each entity are shown below.

[0835] Server Functions and Operations

[0836] The server performs processing using the following hardware and software.

[0837] Hardware: High-performance processor, memory, and storage

[0838] Software: OCR technology (e.g., Tesseract), natural language processing engines (e.g., spaCy, NLTK), emotion recognition engines (e.g., Microsoft Azure Cognitive Services, OpenCV)

[0839] The server first receives the electronic file uploaded by the user and stores it in temporary storage. Next, it uses OCR technology to convert the contents of the electronic file into text data. After conversion, each slide or page in the file is divided into sections and assigned a unique ID. This text data and section information are then stored in a material database.

[0840] The text data stored in the resource database is analyzed using a natural language processing engine. Key keywords and themes are extracted from the title, headings, and body of the text, and the importance of each section is evaluated. The extracted keywords, themes, and importance information are saved in the resource database as metadata.

[0841] The emotion engine analyzes the voice data or facial image data collected when the user searches to recognize the user's emotional state, which is also stored in the database.

[0842] The server analyzes the search prompt sent by the user and extracts relevant themes and keywords using a generative AI model (e.g., BERT, GPT-3). It then generates a search query based on the extracted themes and keywords and searches the document database.

[0843] The generated search query is used to search the document database to select multiple highly relevant documents, extract important slides and sections from each document, and customize the order and content of the documents to be presented based on the user's perceived emotions.

[0844] Device features and operations

[0845] A terminal is a device that a user uses as an interface, and performs processing using the following hardware and software.

[0846] Hardware: microphone, camera, display

[0847] Software: Web browser, communication module

[0848] When a user uploads a document, the terminal transmits the electronic file selected by the user to the server. When a user enters a search prompt, the terminal also collects the user's voice data and facial image data and transmits them to the server.

[0849] User operations

[0850] Users access the system's upload screen, select the electronic file they want to use as reference, and click the upload button. They also enter keywords related to the information they are looking for as prompted on the search screen. They can check the list of suggested materials displayed on the terminal and view or download the materials they are interested in in detail.

[0851] Specific examples

[0852] For example, consider the case where a user uploads a slide set for a conference presentation, "Research Presentation_Machine Learning.ppt." When the user selects a file using the upload screen and clicks the upload button, the device sends the file to the server. The server saves the received file in temporary storage and converts it into text data using OCR technology. The converted text data is divided into sections, and the information is stored in a materials database.

[0853] Next, when a user enters a prompt such as "latest applications of machine learning" into the search screen, the prompt is sent from the device to the server. The server analyzes the prompt and extracts keywords such as "machine learning," "latest," and "application." A search query is generated based on the extracted keywords, and by searching the document database, highly relevant documents are picked up.

[0854] At that time, the device sends the user's voice data and facial image data to the emotion engine, which then recognizes the user's emotions.The server then customizes the presentation order and content of the selected materials based on that emotional information.For example, if the user is feeling impatient, it will prioritize displaying materials that summarize only the main points.

[0855] Finally, the list of suggested materials is sent to the terminal, and the user can check the list and view or download the materials of interest in detail.

[0856] Examples of prompt statements

[0857] The prompt to be input to the generative AI model is as follows:

[0858] "I'm looking for the latest papers related to my research topic."

[0859] "What's the latest news on automotive battery technology?"

[0860] "I'd like to know what to do about this week's economic trends."

[0861] In this way, the system allows users to quickly and efficiently obtain the information they desire.

[0862] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0863] Step 1: Upload and store your materials

[0864] Input: An electronic file selected by the user (e.g. PDF, PPT, etc.)

[0865] Output: Text data and section information are stored in a resource database.

[0866] Specific behavior:

[0867] User: The user accesses the system's upload screen, selects the electronic file they wish to upload, and clicks the upload button.

[0868] Terminal: Sends the file selected by the user to the server. The terminal encodes the file as binary data and sends it via an HTTP request.

[0869] Server: The server stores the received files in temporary storage. It then uses Tesseract OCR technology to convert the file contents into text data. The converted text data is divided into sections for each slide or page, and a unique ID is assigned. This text data and section information are stored in a material database.

[0870] Step 2: Text analysis and feature extraction of the material

[0871] Input: Text data stored in the resource database

[0872] Output: Generates metadata for keywords, themes, and importance information

[0873] Specific behavior:

[0874] Server: The server analyzes the text data stored in the document database using a natural language processing (NLP) engine. Software used includes spaCy and the NLTK library. After analysis, it extracts key keywords and themes from the title, headings, and body of the text. For example, for a document containing the theme of "machine learning," it extracts keywords such as "neural network" and "regression analysis." It then evaluates the importance of the extracted keywords and themes, as well as each section, and stores this information as metadata in the document database.

[0875] Step 3: Emotion recognition by the emotion engine

[0876] Input: User's voice or facial image data

[0877] Output: User's emotional information

[0878] Specific behavior:

[0879] Device: When a user searches, the device collects the user's voice data or facial image data. This is done using a microphone or camera, which records and photographs the data in real time.

[0880] Server: The emotion engine analyzes the collected voice data and facial image data to recognize the user's emotional state. For voice data, it uses the emotion recognition API from Microsoft Azure Cognitive Services, and for facial image data, it uses OpenCV and dlib algorithms to identify whether the user is anxious or calm. This emotional information is stored in a database.

[0881] Step 4: Entering and Parsing Search Prompts

[0882] Input: The search prompt entered by the user

[0883] Output: The generated search query

[0884] Specific behavior:

[0885] User: The user enters a keyword or phrase related to the information they are looking for as a prompt in the search screen. For example, they might enter "latest applications of machine learning."

[0886] Terminal: Sends the entered prompt to the server. The prompt is encoded as string data and sent through an HTTP request.

[0887] Server: The server analyzes the received prompt using a natural language processing model (e.g., GPT-3) to extract relevant themes and keywords, such as "machine learning," "latest," and "applied," and generates a search query based on these.

[0888] Step 5: Search and suggest related materials

[0889] Input: Generated search query and sentiment information

[0890] Output: Related materials picked up

[0891] Specific behavior:

[0892] Server: Using the generated search query, the server searches the resource database. It uses information retrieval algorithms such as TF-IDF and Doc2Vec to match the resource database metadata and find relevant resources.

[0893] Server: Extracts important slides and sections from each presentation. It also customizes the order and content of presentation materials based on the user's perceived emotions. For example, if the user is feeling impatient, it prioritizes short, focused presentation materials.

[0894] Step 6: View the results

[0895] Input: Customized proposal list

[0896] Output: Display the list of materials to the user

[0897] Specific behavior:

[0898] Terminal: The terminal displays the customized list of proposal materials sent from the server to the user. For each material, the title, summary, keywords, and thumbnails of important slides are displayed in a user interface using HTML and CSS.

[0899] User: The user can view the list of proposal documents and find the ones they are interested in for detailed view or download. For example, they can click on the title of a proposal document to access its detailed page and download it in PDF format.

[0900] The above is the flow of program processing for this system.

[0901] (Application example 2)

[0902] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0903] In today's information-saturated society, it is difficult for users to quickly and accurately obtain the specific information they are looking for. In particular, there is no method to efficiently search through materials stored in electronic files and provide appropriate information based on the user's emotions and situation. Furthermore, online shopping sites are in need of a system that can perform personalized product searches that take the user's emotions into account.

[0904] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0905] In this invention, the server includes means for receiving electronic files of reference materials, means for converting the contents of the received electronic files into text data using optical character recognition technology, means for analyzing the text data using a natural language processing engine to extract keywords and themes and generate metadata, and means for recognizing the user's emotions and customizing search results based on the user's emotional state. This enables the user to quickly and accurately obtain the information they are looking for, and further enables the provision of personalized information according to the user's emotions.

[0906] "Electronic reference files" are digital files that store information uploaded by users to the system, including formats such as PDF and PPT.

[0907] "Optical character recognition technology" is a technology that extracts text data from image data, and is also known as OCR (Optical Character Recognition).

[0908] "Text data" refers to digital data that contains textual information and has been converted into a format that can be used for analysis and searching.

[0909] A "natural language processing engine" is a computer program for understanding and analyzing human language, and is used to extract keywords and themes from text data.

[0910] "Metadata" is data that contains information about data, such as keywords, themes, and importance of text data stored in a material database.

[0911] A "database" is a collection of data that is stored and managed electronically and that can be searched and retrieved by users.

[0912] A "search prompt" is a keyword or phrase that a user uses to input the information they are seeking and is used in the system's search function.

[0913] "Emotional state" refers to the user's current emotional state, which the system recognizes by analyzing voice data and image data.

[0914] "Personalization" refers to adapting and customizing information based on a user's individual characteristics and feelings.

[0915] "Important sections" refer to parts or slides that are considered to be particularly important in the materials, and are extracted to quickly provide users with the information they need.

[0916] This invention is a system that recognizes a user's emotions and customizes search results based on a specific emotional state. This system is particularly applicable to online shopping sites, where users not only input images and keywords of the items they want, but also recognize emotions from facial images and voice, and suggest products that best suit those emotions. This system is primarily composed of three entities: a server, a terminal, and the user.

[0917] Server configuration and roles

[0918] The server has the following functions:

[0919] 1. Receiving and processing electronic files: Electronic files uploaded by users are received and converted into text data using OCR technology. This text data is then analyzed using a natural language processing engine to extract keywords and themes, generating metadata, and storing it in the material database.

[0920] 2. Emotion recognition: It has the function of recognizing the user's emotions, and analyzes the voice data and facial image data collected by the emotion engine to identify the user's emotional state.

[0921] 3. Search prompt analysis and customization: Analyze the search prompt received from the user and extract relevant keywords, then customize the search results based on the user's emotional state.

[0922] 4. Product Search and Recommendation: The generated search query is used to search the document database to find relevant products, and then personalized product recommendations are made based on the user's emotional state.

[0923] Device configuration and role

[0924] The terminal has the following functions:

[0925] 1. Uploading an electronic file: The user accesses the upload screen of the system, selects the electronic file they want to use as reference, and clicks the upload button.

[0926] 2. Entering a search prompt: The user enters an image or keyword about the material or product they want as a prompt.

[0927] 3. Emotion data collection: The emotion engine collects the user's voice data and facial image data and sends them to the server.

[0928] 4. Display search results: Display a customized list of suggested products and key information about each product to the user.

[0929] User operations

[0930] The user uses the system in the following steps:

[0931] 1. Uploading an electronic file: The user selects the electronic file they want to use as reference from the upload screen on their device and clicks the upload button. The device then sends the file to the server.

[0932] 2. Enter a search prompt: Enter an image or keyword of the material or product you want as a prompt. For example, enter "latest smartphone."

[0933] 3. Emotion recognition: The camera captures the user's facial image and voice, and the emotion engine analyzes the data to identify the user's emotional state. For example, it recognizes the emotion of "anxiety."

[0934] 4. Review search results: Review the customized list of suggested products sent from the server, and proceed to the detailed view or purchase process for the products that interest you.

[0935] Specific examples

[0936] For example, if a user types in "latest smartphone" and the emotion "impatient" is recognized from an image captured by the camera, the generated search prompt might look like this:

[0937] Prompt statement:

[0938] The latest smartphones with a feeling of anxiety

[0939] Based on this prompt, the server will provide quick and focused results, such as simpler, more readily available models and items that are in stock and ready to buy.

[0940] In this way, the system provides optimal products according to the user's emotional state and enables the user to quickly and efficiently obtain the information they are looking for.

[0941] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0942] Step 1:

[0943] The user accesses the system's upload screen, selects the electronic file they want to use as reference, and clicks the upload button. The terminal then sends this electronic file to the server. The input is the electronic file selected by the user, and the output is the transfer of the electronic file to the server.

[0944] Step 2:

[0945] The server stores the received electronic files and converts the file contents into text data using optical character recognition (OCR). The input is an electronic file, and the output is text data. This text data is organized into sections that separate each slide or page, and stored in a materials database.

[0946] Step 3:

[0947] The server analyzes the text data stored in the resource database using a natural language processing (NLP) engine, extracting key keywords and themes from titles, headings, and main text content. The input is text data, and the output is metadata (extracted keywords, themes, and importance information). This metadata is stored in the resource database.

[0948] Step 4:

[0949] The user enters a search prompt, and the terminal sends the prompt to the server. The input is the prompt sentence entered by the user, and the output is the transfer of the prompt sentence to the server.

[0950] Step 5:

[0951] The server analyzes the received search prompt and extracts relevant keywords. The input is the search prompt sentence and the output is the extracted keywords. The server then generates a search query based on these keywords.

[0952] Step 6:

[0953] The terminal transmits the user's voice data and facial image data to the emotion engine, which analyzes the data and recognizes the user's emotional state. The input is the voice data and facial image data, and the output is the recognized emotional state.

[0954] Step 7:

[0955] The server uses the generated search query to search a database of materials and pick out highly relevant materials and products. It also customizes search results based on the user's emotional state. For example, for a user who is feeling impatient, it prioritizes displaying short, focused materials and products. The input is the search query and the user's emotional state, and the output is customized search results.

[0956] Step 8:

[0957] The server sends customized proposal materials and product lists to the terminal. The terminal displays this information to the user. The input is the customized search results sent from the server, and the output is the display of the search results to the user. The user can view details of the materials or products they are interested in, download them, or proceed to purchase them.

[0958] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0959] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0960] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0961] [Third embodiment]

[0962] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0963] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0964] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0965] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0966] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0967] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0968] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0969] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0970] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0971] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0972] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0973] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0974] The present invention is a system for efficiently searching and suggesting reference materials, particularly those stored as electronic files, and automatically processing them to enable users to quickly and accurately obtain the information they desire.

[0975] This system is mainly composed of three entities: a server, a terminal, and a user. The specific operation of each is explained below.

[0976] Uploading and storing materials

[0977] User: First, the user accesses the system's upload screen, selects the electronic file (e.g., PDF, PPT, etc.) they want to use as reference, and clicks the upload button.

[0978] Terminal: Sends the file selected by the user to the server.

[0979] Server: The received file is stored in temporary storage. Then, using OCR (Optical Character Recognition) technology, the contents of the uploaded file are converted into text data. After conversion, each slide or page in the file is divided into sections and assigned a unique ID. This text data and section information are stored in a material database.

[0980] Text analysis and feature extraction of materials

[0981] Server: Analyzes text data stored in the resource database using a natural language processing (NLP) engine. Extracts key keywords and themes from titles, headings, and main text, and evaluates the importance of each section. The extracted keywords, themes, and importance information are stored in the resource database as metadata.

[0982] Entering and parsing search prompts

[0983] User: Enter the image or keywords of the desired material as a prompt.

[0984] Terminal: Sends the entered prompt to the server.

[0985] Server: Analyzes the received prompts, extracts relevant themes and keywords, and generates search queries based on the extracted themes and keywords.

[0986] Search and suggest related materials

[0987] Server: Searches the resource database using the generated search query. It compares the metadata in the resource database to select multiple highly relevant resources. It then extracts important slides and sections from each of the selected resources and generates summary information for each (title, keywords, importance, etc.). It then sends a list of the selected resources and key sections to the user's device.

[0988] Displaying the results

[0989] Terminal: Displays the user a list of proposal materials and the important slides and sections of each material. For each material, the title, summary, keywords, and thumbnails of important slides are displayed.

[0990] Users: Review the proposed materials and, if interested, view details or download them.

[0991] Specific examples

[0992] For example, consider the case where a user uploads a slide set for a conference presentation, "Research Presentation_Machine Learning.ppt." When the user selects a file using the upload screen and clicks the upload button, the device sends the file to the server. The server saves the received file in temporary storage and converts it into text data using OCR technology. The converted text data is divided into sections, and the information is stored in a materials database.

[0993] Next, when a user enters a prompt such as "latest applications of machine learning" into the search screen, the prompt is sent from the device to the server. The server analyzes the prompt and extracts keywords such as "machine learning," "latest," and "application." A search query is generated based on the extracted keywords, and by searching the document database, highly relevant documents are picked up.

[0994] Important slides (e.g., slides 3 and 7) are extracted from the selected materials, and summary information is generated for each. This information is sent to the user's device and displayed as a list of proposed materials.

[0995] Users can select the materials they want to know more about from the suggested list and view or download them, allowing them to obtain the information they are looking for quickly and efficiently.

[0996] Thus, the present invention is significantly more efficient than existing document retrieval systems and can quickly provide information that meets the user's needs.

[0997] The processing flow will be explained below.

[0998] Step 1: Upload your materials

[0999] User: The user accesses the system's upload screen, selects the electronic file (e.g., PDF, PPT, etc.) they want to use as reference, and clicks the upload button.

[1000] Step 2: Send the file

[1001] Terminal: Sends the file selected by the user to the server.

[1002] Step 3: Save the file

[1003] Server: Stores received files in temporary storage.

[1004] Step 4: OCR processing

[1005] Server: Launches an OCR (optical character recognition) engine and converts the contents of the uploaded file into text data.

[1006] Step 5: Sectioning

[1007] Server: Each slide or page in the text data file is divided into sections, and a unique ID is assigned to each section.

[1008] Step 6: Store in the database

[1009] Server: Stores the converted text data and section information in the material database.

[1010] Step 7: Natural Language Processing Analysis

[1011] Server: Analyzes the text data stored in the material database using a natural language processing (NLP) engine.

[1012] Step 8: Extracting keywords and themes

[1013] Server: Extracts key keywords and themes from titles, headings, and body content, and rates the importance of each section.

[1014] Step 9: Generate Metadata

[1015] Server: Generates extracted keywords, themes, and importance information as metadata and adds them to the material database.

[1016] Step 10: Enter the search prompt

[1017] User: Accesses the system's search screen and enters the image or keywords of the desired material as prompted.

[1018] Step 11: Sending prompts

[1019] Terminal: Sends the entered prompt to the server.

[1020] Step 12: Parse prompts

[1021] Server: Parses the received prompts and extracts relevant themes and keywords.

[1022] Step 13: Generate a search query

[1023] Server: Generates search queries based on the extracted themes and keywords.

[1024] Step 14: Search the documentation database

[1025] Server: Searches the resource database using the generated search query.

[1026] Step 15: Identify relevant materials

[1027] Server: Compares the metadata in the material database and selects multiple highly relevant materials.

[1028] Step 16: Extracting important sections

[1029] Server: Extracts key slides and sections from each document.

[1030] Step 17: Generate summary information

[1031] Server: Generates summary information (title, keywords, importance, etc.) for the extracted important slides and sections.

[1032] Step 18: Submit your proposal list

[1033] Server: Sends the list of proposal materials and the list of important sections to the user's terminal.

[1034] Step 19: View the results

[1035] Terminal: Displays the user a list of proposal materials and key slides and sections of each material.

[1036] Step 20: View details and download

[1037] Users: Review the suggested materials and select the ones that interest them for detailed viewing or download.

[1038] In this way, the system provides the user with the materials they require efficiently and quickly through a series of steps.

[1039] Example 1

[1040] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1041] Conventional reference material search systems have limitations in their ability to efficiently search and suggest the contents of electronic files, making it difficult for users to quickly and accurately obtain the information they need. Furthermore, they lacked the technology to improve the accuracy of suggesting highly relevant materials and the ability to extract and display important sections of materials. This resulted in the problem of users spending a great deal of time and effort finding the information they were looking for.

[1042] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1043] In this invention, the server includes: means for receiving electronic files of reference materials; means for converting the contents of the received electronic files into text data using optical character recognition technology; means for analyzing the text data with a natural language processing engine to extract keywords and themes and generate metadata; means for storing the metadata in a database; means for receiving search prompts and extracting relevant keywords from the prompts; means for searching the database based on the extracted keywords to select relevant materials; means for extracting important sections from the selected materials and providing them to the user; means for assigning unique IDs to the extracted sections and evaluating their importance; means for extracting themes from the prompts using a machine learning model; means for recommending highly relevant materials based on the metadata of the materials stored in the database; means for generating summary information of the relevant materials and transmitting it to the user terminal; and means for analyzing the search prompts and generating search queries. This allows users to quickly and efficiently obtain the information they need and effectively search for and suggest highly relevant materials.

[1044] "Electronic reference files" refers to materials and documents stored in digital formats such as PDF and PPT.

[1045] The "means for receiving" refers to a function for acquiring an electronic file when the server receives an uploaded file from a user.

[1046] "Optical character recognition technology" is a technology called OCR (Optical Character Recognition), which analyzes characters in an image and converts them into text data.

[1047] "Text data" refers to data in which only character information is extracted from the contents of an electronic file and saved in text format.

[1048] A "natural language processing engine" is a computer science technology for analyzing text data to extract keywords and themes.

[1049] "Metadata" refers to data that includes keywords, themes, section importance information, and the like extracted from text data.

[1050] A "database" is a system for efficiently storing, retrieving, and managing various types of data.

[1051] A "search prompt" is a keyword or phrase that a user enters to search for the information they need.

[1052] "Means for extracting relevant keywords" refers to a function for extracting important keywords from a search prompt.

[1053] A "search query" is a specific search condition generated to search a database.

[1054] "Means for picking out relevant materials" refers to a function for selecting relevant materials from a database based on a search query.

[1055] "Means for extracting important sections" refers to a function for selecting particularly important parts from the picked-up material.

[1056] A "unique ID" is an identifier assigned to each section to uniquely identify it.

[1057] The "means for evaluating the importance" refers to a function for quantitatively evaluating the importance of the extracted section.

[1058] A "machine learning model" is an algorithm or technique that learns from data and makes predictions or classifications based on that data.

[1059] "Means for recommending highly relevant materials" refers to a function for selecting and suggesting materials that best fit the user's search prompt.

[1060] The "means for generating summary information and transmitting it to the user terminal" refers to a function for generating a summary of the picked-up material and transmitting it to the user terminal.

[1061] The present invention is a system for efficiently searching and suggesting reference materials. This system automatically processes materials stored as electronic files, enabling users to quickly and accurately obtain the information they desire. The system is primarily composed of three entities: a server, a terminal, and a user. The specific operation of each is described below.

[1062] Uploading and storing materials

[1063] Users access the system's upload screen, select the electronic file they want to use as reference (e.g. PDF, PPT, etc.), and click the upload button. The terminal sends the file selected by the user to the server. The server stores the received file in temporary storage. The server then uses OCR (optical character recognition) technology to convert the contents of the uploaded file into text data. Tesseract OCR is used for the conversion. After conversion, each slide or page in the file is divided into sections and assigned a unique ID. This text data and section information are stored in a material database. The material database uses MySQL.

[1064] Text analysis and feature extraction of materials

[1065] The server analyzes the text data stored in the document database using a natural language processing (NLP) engine. The NLP engine used is SpaCy. It extracts key keywords and themes from titles, headings, and body content, and uses the TF-IDF algorithm to evaluate the importance of each section. The extracted keywords, themes, and importance information are stored in the document database as metadata.

[1066] Entering and parsing search prompts

[1067] The user enters an image of the desired material or keywords as a prompt into the search screen. The device sends the entered prompt to the server. The server analyzes the received prompt and extracts related themes and keywords. The BERT model is used to analyze the prompt. A search query is generated based on the extracted keywords.

[1068] Search and suggest related materials

[1069] The server searches the document database using the generated search query. It compares the results with the metadata in the document database to select multiple highly relevant documents. It calculates a relevance score and selects the top documents. It extracts the most important slides and sections from each of the selected documents and generates summary information (title, keywords, importance, etc.) for each. The generated summary information is sent from the server to the user's device in JSON format.

[1070] Displaying the results

[1071] The terminal displays a list of proposed materials and the important slides and sections of each material to the user. For each material, the title, summary, keywords, and thumbnails of important slides are displayed. HTML / CSS and JavaScript are used to display the user interface. The user can check the proposed materials and, if interested, can view details or download them. Downloads are performed using an HTTP GET request.

[1072] Specific examples

[1073] For example, consider the case where a user uploads a slide set for a conference presentation, "Research Presentation_Machine Learning.ppt." When the user selects a file using the upload screen and clicks the upload button, the device sends the file to the server. The server saves the received file in temporary storage and converts it into text data using OCR technology. The converted text data is divided into sections, and the information is stored in a materials database.

[1074] Next, if a user enters a prompt such as "latest applications of machine learning" into the search screen, the prompt is sent from the device to the server. The server analyzes the prompt and extracts keywords such as "machine learning," "latest," and "application." A search query is generated based on the extracted keywords, and highly relevant materials are picked out by searching the materials database. Important slides (e.g., slides 3 and 7) are extracted from the picked materials, and summary information is generated for each. This information is sent to the user's device and displayed as a list of suggested materials.

[1075] This system allows users to quickly and efficiently obtain the information they need, significantly improving the efficiency of research and work. As a result, it becomes possible to provide information that meets the user's needs appropriately.

[1076] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1077] Step 1:

[1078] The user accesses the upload screen of the system, selects the electronic file they want to use as reference, and clicks the upload button.

[1079] Input: An electronic file selected by the user (e.g. PDF, PPT, etc.)

[1080] Output: Request to upload selected files

[1081] Step 2:

[1082] The terminal transmits the file selected by the user to the server.

[1083] Input: User upload request and selected electronic file

[1084] Output: A request for an electronic file to be sent to the server

[1085] Step 3:

[1086] The server stores the received file in temporary storage.

[1087] Input: Electronic file sent from the terminal

[1088] Output: Electronic file saved in temporary storage

[1089] Step 4:

[1090] The server uses OCR technology to convert the contents of the uploaded file into text data.

[1091] Input: Electronic files stored in temporary storage

[1092] Output: Converted text data

[1093] Step 5:

[1094] The server divides the text data into sections and assigns each a unique ID.

[1095] Input: Converted text data

[1096] Output: Text data with a unique ID assigned to each section

[1097] Step 6:

[1098] The server stores the section information and text data in a material database.

[1099] Input: Text data divided into sections

[1100] Output: Text data stored in the material database and its section information

[1101] Step 7:

[1102] The server uses a natural language processing engine to analyze the text data stored in the document database, extract key keywords and themes, and evaluate the importance of each section.

[1103] Input: Text data stored in the material database

[1104] Output: Extracted keywords, themes, and importance of each section

[1105] Step 8:

[1106] The server stores the extracted keywords, themes, and importance information as metadata in the document database.

[1107] Input: Extracted keywords, themes, and importance information

[1108] Output: Metadata stored in the material database

[1109] Step 9:

[1110] The user enters an image of the material they want or keywords into the search screen as a prompt.

[1111] Input: Search prompt

[1112] Output: User search request

[1113] Step 10:

[1114] The terminal sends the entered prompt to the server.

[1115] Input: User search request

[1116] Output: Sending a search prompt to the server

[1117] Step 11:

[1118] The server analyzes the prompts it receives and uses machine learning models to extract relevant themes and keywords.

[1119] Input: Submitted search prompt

[1120] Output: Themes and keywords extracted from the prompt

[1121] Step 12:

[1122] The server generates a search query based on the extracted themes and keywords.

[1123] Input: Extracted themes and keywords

[1124] Output: Generated search query

[1125] Step 13:

[1126] The server searches the document database using the generated search query and picks out multiple highly relevant documents.

[1127] Input: Generated search query

[1128] Output: Related materials picked up

[1129] Step 14:

[1130] The server extracts the most important slides and sections from each document and generates a summary.

[1131] Input: Picked up materials

[1132] Output: Summary information (title, keywords, importance, etc.)

[1133] Step 15:

[1134] The server transmits a list of proposal materials including summary information to the user's terminal.

[1135] Input: Summary Information

[1136] Output: A list of proposal materials sent to the user's device

[1137] Step 16:

[1138] The device displays a list of proposal materials and important slides and sections of each material to the user.

[1139] Input: Proposal list sent from the server

[1140] Output: A list of proposals displayed to the user

[1141] Step 17:

[1142] The user can review the proposed materials and, if interested, view them in detail or download them.

[1143] Input: Displayed proposal list

[1144] Output: View detailed view or download document

[1145] Through the above steps, the user can quickly and efficiently obtain the information he or she needs.

[1146] (Application example 1)

[1147] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1148] Factories and industrial sites often contain vast amounts of technical documents and manuals, but it is not easy to quickly and accurately search through them and obtain the information needed. Managing and searching through these documents takes a great deal of time and effort, especially when there are many paper documents and printed materials. This situation reduces work efficiency and impacts productivity. Furthermore, delays in accessing necessary information can lead to delays in equipment maintenance and troubleshooting. To address these challenges, an effective document management and search system is needed.

[1149] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1150] In this invention, the server includes: means for receiving electronic files of reference materials; means for converting the contents of the received electronic files into text data using optical character recognition technology; means for analyzing the text data with a natural language processing engine to extract keywords and themes and generate metadata; means for storing a database for storing the metadata; means for receiving search prompts and extracting relevant keywords from the prompts; means for searching a database based on the extracted keywords to find relevant materials; means for extracting important sections from the found materials and providing them to a user; means for a user to scan paper documents or printed materials with a smartphone and upload the images; means for processing the uploaded images with OCR technology and converting them to text; and means for searching a database using a TF-IDF-based search algorithm to suggest relevant materials in response to a user's search query, enabling factories and industrial sites to quickly and accurately search for necessary materials and provide them to users.

[1151] "Electronic files of reference materials" are technical documents and manuals related to work in factories and industrial sites that are stored in digital format.

[1152] Optical character recognition (OCR) is a technology that scans characters written on paper documents or printed materials and converts them into digital character data.

[1153] "Text data" refers to the text information in electronic files converted using OCR technology and stored in digital format.

[1154] A "natural language processing engine (NLP)" is a program that analyzes text data and automatically extracts important keywords and themes.

[1155] "Metadata" refers to additional information extracted from text data, such as keywords, themes, and importance, which is used to efficiently search and organize data within a database.

[1156] A "database" is a system for centrally managing and storing metadata and related material information.

[1157] A "search prompt" is a keyword or query that a user enters to specifically describe the information they are seeking.

[1158] A "TF-IDF-based search algorithm" is a statistical method for evaluating the importance of documents in a database based on a search query and deriving highly relevant materials.

[1159] "Scanning with a smartphone" refers to the process by which a user uses the smartphone's camera function to convert paper documents or printed materials into digital images.

[1160] "Uploading" is the act of a user transferring a digital image or electronic file to a cloud system such as a server.

[1161] "Important sections" are those parts of the selected material that are particularly relevant and useful to the user.

[1162] The present invention is a document management and search system for use in factories and industrial sites. This system is composed of three main components: a server, a terminal, and a user. The specific operation of each component will be explained below.

[1163] Uploading and storing materials

[1164] User: The user uses their smartphone to scan paper documents or printed materials and convert them into digital images, then selects the scanned image on the application's upload screen and clicks the upload button.

[1165] Terminal: Sends the image file selected by the user to the server.

[1166] Server: The received image files are stored in temporary storage and the contents are converted into text data using OCR (Optical Character Recognition) technology. After conversion, each file is divided into sections and assigned a unique ID. This text data and section information are stored in a material database.

[1167] Text analysis and feature extraction of materials

[1168] Server: Analyzes text data stored in the resource database using a natural language processing (NLP) engine. Extracts key keywords and themes and evaluates the importance of each section. The extracted keywords, themes, and importance information are stored in the resource database as metadata.

[1169] Entering and parsing search prompts

[1170] User: The user enters a search prompt into the application for the required material, for example, "Basic steps in PLC programming."

[1171] Terminal: Sends the entered prompt to the server.

[1172] Server: Analyzes the received prompts, extracts relevant themes and keywords, and generates search queries based on the extracted themes and keywords.

[1173] Search and suggest related materials

[1174] Server: Searches the document database using the generated search query. It uses a TF-IDF-based search algorithm to evaluate the importance of documents in the database and selects highly relevant documents. It then extracts important slides and sections from each of the selected documents and generates summary information for each (title, keywords, importance, etc.). It then sends a list of the selected documents and key sections to the user's device.

[1175] Displaying the results

[1176] Terminal: Displays the user a list of proposed materials and the important sections of each material. For each material, displays the title, summary, keywords, and thumbnails of important sections.

[1177] Users: Review the proposed materials and, if interested, view details or download them.

[1178] The system's hardware mainly requires a smartphone with a camera and a high-performance server. It uses the pytesseract library for OCR technology, and the nltk and scikit-learn libraries for natural language processing. The TF-IDF algorithm is used for data analysis of the documents.

[1179] Specific examples

[1180] For example, consider a user on a factory floor looking for a "Troubleshooting Guide for PLC Programming." The user scans a paper manual with their smartphone and uploads it to the server using an application. Next, they input "Basic Steps for PLC Programming" as a search prompt. The server analyzes this prompt and uses a TF-IDF-based algorithm to search and suggest relevant documents from the database. The user then checks the detailed view of the displayed document list to obtain the information they need.

[1181] The above method makes it possible to quickly and accurately search for and provide users with the necessary documents and information at factories and industrial sites. This system not only improves work efficiency at the site, but also speeds up troubleshooting and maintenance work by providing timely and appropriate information.

[1182] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1183] Step 1:

[1184] A user uses a smartphone to scan paper documents or printed materials and convert them into digital images. Then, they select the scanned image on the application's upload screen and click the upload button. At this point, the input is the scanned image file, and the output is an upload request.

[1185] Step 2:

[1186] The terminal sends the image file selected by the user to the server. The input here is the scanned image file, and the output is the image file sent to the server. This sending process is performed via the Internet.

[1187] Step 3:

[1188] The server stores the received image file in temporary storage. It then uses OCR technology to convert the image file into text data. The input here is the sent image file, and the output is text data. Specifically, it uses the pytesseract library to recognize the text in the image and convert it into text.

[1189] Step 4:

[1190] The server divides the converted text data into sections, assigns unique IDs, and stores them in a database. In this step, the input is text data, and the output is text data in a database divided into sections.

[1191] Step 5:

[1192] The server uses a natural language processing engine (NLP) to analyze the text data stored in the resource database and extract key keywords and themes. The input is the text data in the database, and the output is metadata. Specifically, it uses NLP technology to extract important words and phrases from the text and stores them as metadata.

[1193] Step 6:

[1194] The user enters a search prompt for the required material on the application search screen, for example, a keyword such as "Basic steps of PLC programming." The input of this step is the search prompt, and the output is a prompt transmission request from the terminal to the server.

[1195] Step 7:

[1196] The terminal sends the entered prompt to the server, where the input is the search prompt entered by the user and the output is a prompt transmission request to the server.

[1197] Step 8:

[1198] The server analyzes the prompts it receives, extracts relevant themes and keywords, and generates a search query based on that information. The input is the prompt, and the output is the generated search query. This analysis is performed using a generative AI model.

[1199] Step 9:

[1200] The server uses the generated search query to search the resource database, where the input is the search query and the output is a list of relevant resources. A TF-IDF-based search algorithm is used to evaluate and select relevant documents.

[1201] Step 10:

[1202] The server extracts important sections from each of the documents picked up and generates summary information (title, keywords, importance, etc.) for each. The input is a list of documents, and the output is a list of proposed documents with summary information.

[1203] Step 11:

[1204] The server sends the list of proposal materials and information on important sections of each material to the user's terminal. The input is the list of proposal materials with summary information, and the output is a transmission request to the user's terminal.

[1205] Step 12:

[1206] The terminal displays the list of proposal materials and important sections of each material to the user. The input is the list of proposal materials sent from the server, and the output is the material information displayed to the user.

[1207] Step 13:

[1208] The user checks the proposed materials and, if interested, can view details or download them. The input is the material information displayed to the user, and the output is the downloaded detailed materials.

[1209] This series of processes makes it possible to quickly and accurately search for necessary materials and information at factories and industrial sites and provide them to users.

[1210] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1211] The present invention is a system for efficiently searching electronic files of reference materials and quickly providing users with the information they desire. Furthermore, the present invention has the ability to recognize a user's emotions and customize search results based on those emotions. This system is specifically designed to automatically process materials stored as electronic files, enabling users to quickly and accurately obtain the information they desire.

[1212] This system is mainly composed of three entities: a server, a terminal, and a user. The specific operation of each is explained below.

[1213] Uploading and storing materials

[1214] User: The user accesses the system's upload screen, selects the electronic file (e.g., PDF, PPT, etc.) they want to use as reference, and clicks the upload button.

[1215] Terminal: Sends the file selected by the user to the server.

[1216] Server: The received file is stored in temporary storage. Then, using OCR (Optical Character Recognition) technology, the contents of the uploaded file are converted into text data. After conversion, each slide or page in the file is divided into sections and assigned a unique ID. This text data and section information are stored in a material database.

[1217] Text analysis and feature extraction of materials

[1218] Server: Analyzes text data stored in the resource database using a natural language processing (NLP) engine. Extracts key keywords and themes from titles, headings, and main text, and evaluates the importance of each section. The extracted keywords, themes, and importance information are stored in the resource database as metadata.

[1219] Emotion recognition by emotion engine

[1220] On the device: When a user searches, the emotion engine collects the user's voice data or facial image data.

[1221] Server: The emotion engine analyzes the collected data and recognizes the user's emotional state, for example, identifying whether the user is anxious or calm.

[1222] Entering and parsing search prompts

[1223] User: Enter the image or keywords of the desired material as a prompt.

[1224] Terminal: Sends the entered prompt to the server.

[1225] Server: Analyzes the received prompts, extracts relevant themes and keywords, and generates search queries based on the extracted themes and keywords.

[1226] Search and suggest related materials

[1227] Server: Searches the resource database using the generated search query, compares it with the resource database metadata, and selects multiple highly relevant resources.

[1228] Server: Extracts important slides and sections from each presentation and customizes the order and content of the presentation based on the user's perceived emotions. For example, prioritize short, concise presentations for impatient users.

[1229] Displaying the results

[1230] Terminal: Presents users with a customized list of presentation materials and key slides and sections for each material. For each presentation, it displays the title, summary, keywords, and thumbnails of key slides.

[1231] Users: Review the proposed materials and, if interested, view details or download them.

[1232] Specific examples

[1233] For example, consider the case where a user uploads a slide set for a conference presentation, "Research Presentation_Machine Learning.ppt." When the user selects a file using the upload screen and clicks the upload button, the device sends the file to the server. The server saves the received file in temporary storage and converts it into text data using OCR technology. The converted text data is divided into sections, and the information is stored in a materials database.

[1234] Next, when a user enters a prompt such as "latest applications of machine learning" into the search screen, the prompt is sent from the device to the server. The server analyzes the prompt and extracts keywords such as "machine learning," "latest," and "application." A search query is generated based on the extracted keywords, and by searching the document database, highly relevant documents are picked up.

[1235] At that time, the device sends the user's voice data and facial image data to the emotion engine, which then recognizes the user's emotions.The server then customizes the presentation order and content of the selected materials based on that emotional information.For example, if the user is feeling impatient, it will prioritize displaying materials that summarize only the main points.

[1236] Finally, the list of suggested materials is sent to the terminal, where the user can review the list and view or download the materials they are interested in. In this way, the system provides the most appropriate materials according to the user's emotional state, enabling the user to quickly and efficiently obtain the information they are looking for.

[1237] The processing flow will be explained below.

[1238] Step 1: Upload your materials

[1239] User: The user accesses the system's upload screen, selects the electronic file (e.g., PDF, PPT, etc.) they want to use as reference, and clicks the upload button.

[1240] Step 2: Send the file

[1241] Terminal: Sends the file selected by the user to the server.

[1242] Step 3: Save the file

[1243] Server: Stores received files in temporary storage.

[1244] Step 4: OCR processing

[1245] Server: Launches an OCR (optical character recognition) engine and converts the contents of the uploaded file into text data.

[1246] Step 5: Sectioning

[1247] Server: Each slide or page in the text data file is divided into sections, and a unique ID is assigned to each section.

[1248] Step 6: Store in the database

[1249] Server: Stores the converted text data and section information in the material database.

[1250] Step 7: Natural Language Processing Analysis

[1251] Server: Analyzes the text data stored in the material database using a natural language processing (NLP) engine.

[1252] Step 8: Extracting keywords and themes

[1253] Server: Extracts key keywords and themes from titles, headings, and body content, and rates the importance of each section.

[1254] Step 9: Generate Metadata

[1255] Server: Generates extracted keywords, themes, and importance information as metadata and adds them to the material database.

[1256] Step 10: Enter the search prompt

[1257] User: Accesses the system's search screen and enters the image or keywords of the desired material as prompted.

[1258] Step 11: Sending prompts

[1259] Terminal: Sends the entered prompt to the server.

[1260] Step 12: Parse prompts

[1261] Server: Parses the received prompts and extracts relevant themes and keywords.

[1262] Step 13: Generate a search query

[1263] Server: Generates search queries based on the extracted themes and keywords.

[1264] Step 14: Collect emotion data

[1265] Device: In parallel with entering the search prompt, the user's voice data and facial image data are sent to the emotion engine.

[1266] Step 15: Analyze the emotion data

[1267] Server: The emotion engine analyzes the collected voice data and facial image data to recognize the user's emotional state, for example, determining whether they are "anxious" or "calm."

[1268] Step 16: Search the documentation database

[1269] Server: Searches the resource database using the generated search query.

[1270] Step 17: Identify relevant materials

[1271] Server: Compares the metadata in the material database and selects multiple highly relevant materials.

[1272] Step 18: Extracting important sections

[1273] Server: Extracts key slides and sections from each document.

[1274] Step 19: Emotional Customization

[1275] Server: Customize the presentation order and content of the selected materials based on the user's recognized emotional state. For example, prioritize displaying materials that summarize the main points to a user who is feeling impatient.

[1276] Step 20: Submit your proposal list

[1277] Server: Sends a customized list of proposal materials and a list of important sections to the user's device.

[1278] Step 21: View the results

[1279] Terminal: Displays the user a list of proposal materials and key slides and sections of each material.

[1280] Step 22: View details and download

[1281] Users: Review the suggested materials and select those that interest them for detailed viewing or download.

[1282] Example 2

[1283] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1284] Conventional document search systems have difficulty in quickly and accurately providing the information users are looking for. Furthermore, because they provide uniform search results without considering the user's emotions or state, it is difficult to provide information that meets the user's needs. Furthermore, there are often too many search results, which can lead to important information being overlooked, resulting in a decrease in user satisfaction.

[1285] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1286] In this invention, the server includes means for receiving electronic files of reference materials, means for converting the contents of the received electronic files into text data using optical character recognition technology, means for analyzing the text data with a natural language processing engine to extract keywords and themes and generate metadata, means for analyzing audio data or image data and recognizing the user's emotions, means for searching a database based on the extracted keywords and the recognized emotions to select relevant materials, and means for extracting important sections from the selected materials and providing them to the user. This makes it possible to provide appropriate information according to the user's emotional state, enabling the user to quickly and accurately obtain the information they are looking for.

[1287] "Reference materials" refers to electronic files that users use to obtain information, such as PDFs and PPTs.

[1288] "Optical character recognition technology" is a technology that reads characters from image data and converts them into text data.

[1289] "Text data" refers to data that contains only text information and that has been converted using optical character recognition technology.

[1290] "Natural language processing engine" is a general term for software and algorithms that analyze text data and extract keywords and themes.

[1291] "Metadata" refers to data attribute information, such as keywords, themes, and relevance evaluation information extracted from text data.

[1292] A "search prompt" refers to the keywords or phrases that a user enters about the search target.

[1293] "Emotion recognition" is a technology that analyzes voice data and image data to identify a user's emotional state.

[1294] "Database" refers to an information system that stores data in a structured way to facilitate retrieval and access.

[1295] "Key sections" are those parts of the document that are assessed as being particularly important and include key points and summary information.

[1296] A "thumbnail" is a small version of a document or image that allows you to easily check the content.

[1297] A "machine learning model" refers to an algorithm or model that learns patterns based on data and makes predictions and classifications.

[1298] The present invention is a system for efficiently searching electronic files of reference materials and quickly providing users with the information they desire. Furthermore, the present invention has the ability to recognize users' emotions and customize search results based on those emotions. This system is specifically designed to automatically process materials stored as electronic files, enabling users to quickly and accurately obtain the information they desire.

[1299] This system is mainly composed of three entities: a server, a terminal, and a user. The specific operations performed by each entity are shown below.

[1300] Server Functions and Operations

[1301] The server performs processing using the following hardware and software.

[1302] Hardware: High-performance processor, memory, and storage

[1303] Software: OCR technology (e.g., Tesseract), natural language processing engines (e.g., spaCy, NLTK), emotion recognition engines (e.g., Microsoft Azure Cognitive Services, OpenCV)

[1304] The server first receives the electronic file uploaded by the user and stores it in temporary storage. Next, it uses OCR technology to convert the contents of the electronic file into text data. After conversion, each slide or page in the file is divided into sections and assigned a unique ID. This text data and section information are then stored in a material database.

[1305] The text data stored in the resource database is analyzed using a natural language processing engine. Key keywords and themes are extracted from the title, headings, and body of the text, and the importance of each section is evaluated. The extracted keywords, themes, and importance information are saved in the resource database as metadata.

[1306] The emotion engine analyzes the voice data or facial image data collected when the user searches to recognize the user's emotional state, which is also stored in the database.

[1307] The server analyzes the search prompt sent by the user and extracts relevant themes and keywords using a generative AI model (e.g., BERT, GPT-3). It then generates a search query based on the extracted themes and keywords and searches the document database.

[1308] The generated search query is used to search the document database to select multiple highly relevant documents, extract important slides and sections from each document, and customize the order and content of the documents to be presented based on the user's perceived emotions.

[1309] Device features and operations

[1310] A terminal is a device that a user uses as an interface, and performs processing using the following hardware and software.

[1311] Hardware: microphone, camera, display

[1312] Software: Web browser, communication module

[1313] When a user uploads a document, the terminal transmits the electronic file selected by the user to the server. When a user enters a search prompt, the terminal also collects the user's voice data and facial image data and transmits them to the server.

[1314] User operations

[1315] Users access the system's upload screen, select the electronic file they want to use as reference, and click the upload button. They also enter keywords related to the information they are looking for as prompted on the search screen. They can check the list of suggested materials displayed on the terminal and view or download the materials they are interested in in detail.

[1316] Specific examples

[1317] For example, consider the case where a user uploads a slide set for a conference presentation, "Research Presentation_Machine Learning.ppt." When the user selects a file using the upload screen and clicks the upload button, the device sends the file to the server. The server saves the received file in temporary storage and converts it into text data using OCR technology. The converted text data is divided into sections, and the information is stored in a materials database.

[1318] Next, when a user enters a prompt such as "latest applications of machine learning" into the search screen, the prompt is sent from the device to the server. The server analyzes the prompt and extracts keywords such as "machine learning," "latest," and "application." A search query is generated based on the extracted keywords, and by searching the document database, highly relevant documents are picked up.

[1319] At that time, the device sends the user's voice data and facial image data to the emotion engine, which then recognizes the user's emotions.The server then customizes the presentation order and content of the selected materials based on that emotional information.For example, if the user is feeling impatient, it will prioritize displaying materials that summarize only the main points.

[1320] Finally, the list of suggested materials is sent to the terminal, and the user can check the list and view or download the materials of interest in detail.

[1321] Examples of prompt statements

[1322] The prompt to be input to the generative AI model is as follows:

[1323] "I'm looking for the latest papers related to my research topic."

[1324] "What's the latest news on automotive battery technology?"

[1325] "I'd like to know what to do about this week's economic trends."

[1326] In this way, the system allows users to quickly and efficiently obtain the information they desire.

[1327] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1328] Step 1: Upload and store your materials

[1329] Input: An electronic file selected by the user (e.g. PDF, PPT, etc.)

[1330] Output: Text data and section information are stored in a resource database.

[1331] Specific behavior:

[1332] User: The user accesses the system's upload screen, selects the electronic file they wish to upload, and clicks the upload button.

[1333] Terminal: Sends the file selected by the user to the server. The terminal encodes the file as binary data and sends it via an HTTP request.

[1334] Server: The server stores the received files in temporary storage. It then uses Tesseract OCR technology to convert the file contents into text data. The converted text data is divided into sections for each slide or page, and a unique ID is assigned. This text data and section information are stored in a material database.

[1335] Step 2: Text analysis and feature extraction of the material

[1336] Input: Text data stored in the resource database

[1337] Output: Generates metadata for keywords, themes, and importance information

[1338] Specific behavior:

[1339] Server: The server analyzes the text data stored in the document database using a natural language processing (NLP) engine. Software used includes spaCy and the NLTK library. After analysis, it extracts key keywords and themes from the title, headings, and body of the text. For example, for a document containing the theme of "machine learning," it extracts keywords such as "neural network" and "regression analysis." It then evaluates the importance of the extracted keywords and themes, as well as each section, and stores this information as metadata in the document database.

[1340] Step 3: Emotion recognition by the emotion engine

[1341] Input: User's voice or facial image data

[1342] Output: User's emotional information

[1343] Specific behavior:

[1344] Device: When a user searches, the device collects the user's voice data or facial image data. This is done using a microphone or camera, which records and photographs the data in real time.

[1345] Server: The emotion engine analyzes the collected voice data and facial image data to recognize the user's emotional state. For voice data, it uses the emotion recognition API from Microsoft Azure Cognitive Services, and for facial image data, it uses OpenCV and dlib algorithms to identify whether the user is anxious or calm. This emotional information is stored in a database.

[1346] Step 4: Entering and Parsing Search Prompts

[1347] Input: The search prompt entered by the user

[1348] Output: The generated search query

[1349] Specific behavior:

[1350] User: The user enters a keyword or phrase related to the information they are looking for as a prompt in the search screen. For example, they might enter "latest applications of machine learning."

[1351] Terminal: Sends the entered prompt to the server. The prompt is encoded as string data and sent through an HTTP request.

[1352] Server: The server analyzes the received prompt using a natural language processing model (e.g., GPT-3) to extract relevant themes and keywords, such as "machine learning," "latest," and "applied," and generates a search query based on these.

[1353] Step 5: Search and suggest related materials

[1354] Input: Generated search query and sentiment information

[1355] Output: Related materials picked up

[1356] Specific behavior:

[1357] Server: Using the generated search query, the server searches the resource database. It uses information retrieval algorithms such as TF-IDF and Doc2Vec to match the resource database metadata and find relevant resources.

[1358] Server: Extracts important slides and sections from each presentation. It also customizes the order and content of presentation materials based on the user's perceived emotions. For example, if the user is feeling impatient, it prioritizes short, focused presentation materials.

[1359] Step 6: View the results

[1360] Input: Customized proposal list

[1361] Output: Display the list of materials to the user

[1362] Specific behavior:

[1363] Terminal: The terminal displays the customized list of proposal materials sent from the server to the user. For each material, the title, summary, keywords, and thumbnails of important slides are displayed in a user interface using HTML and CSS.

[1364] User: The user can view the list of proposal documents and find the ones they are interested in for detailed view or download. For example, they can click on the title of a proposal document to access its detailed page and download it in PDF format.

[1365] The above is the flow of program processing for this system.

[1366] (Application example 2)

[1367] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1368] In today's information-saturated society, it is difficult for users to quickly and accurately obtain the specific information they are looking for. In particular, there is no method to efficiently search through materials stored in electronic files and provide appropriate information based on the user's emotions and situation. Furthermore, online shopping sites are in need of a system that can perform personalized product searches that take the user's emotions into account.

[1369] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1370] In this invention, the server includes means for receiving electronic files of reference materials, means for converting the contents of the received electronic files into text data using optical character recognition technology, means for analyzing the text data using a natural language processing engine to extract keywords and themes and generate metadata, and means for recognizing the user's emotions and customizing search results based on the user's emotional state. This enables the user to quickly and accurately obtain the information they are looking for, and further enables the provision of personalized information according to the user's emotions.

[1371] "Electronic reference files" are digital files that store information uploaded by users to the system, including formats such as PDF and PPT.

[1372] "Optical character recognition technology" is a technology that extracts text data from image data, and is also known as OCR (Optical Character Recognition).

[1373] "Text data" refers to digital data that contains textual information and has been converted into a format that can be used for analysis and searching.

[1374] A "natural language processing engine" is a computer program for understanding and analyzing human language, and is used to extract keywords and themes from text data.

[1375] "Metadata" is data that contains information about data, such as keywords, themes, and importance of text data stored in a material database.

[1376] A "database" is a collection of data that is stored and managed electronically and that can be searched and retrieved by users.

[1377] A "search prompt" is a keyword or phrase that a user uses to input the information they are seeking and is used in the system's search function.

[1378] "Emotional state" refers to the user's current emotional state, which the system recognizes by analyzing voice data and image data.

[1379] "Personalization" refers to adapting and customizing information based on a user's individual characteristics and feelings.

[1380] "Important sections" refer to parts or slides that are considered to be particularly important in the materials, and are extracted to quickly provide users with the information they need.

[1381] This invention is a system that recognizes a user's emotions and customizes search results based on a specific emotional state. This system is particularly applicable to online shopping sites, where users not only input images and keywords of the items they want, but also recognize emotions from facial images and voice, and suggest products that best suit those emotions. This system is primarily composed of three entities: a server, a terminal, and the user.

[1382] Server configuration and roles

[1383] The server has the following functions:

[1384] 1. Receiving and processing electronic files: Electronic files uploaded by users are received and converted into text data using OCR technology. This text data is then analyzed using a natural language processing engine to extract keywords and themes, generating metadata, and storing it in the material database.

[1385] 2. Emotion recognition: It has the function of recognizing the user's emotions, and analyzes the voice data and facial image data collected by the emotion engine to identify the user's emotional state.

[1386] 3. Search prompt analysis and customization: Analyze the search prompt received from the user and extract relevant keywords, then customize the search results based on the user's emotional state.

[1387] 4. Product Search and Recommendation: The generated search query is used to search the document database to find relevant products, and then personalized product recommendations are made based on the user's emotional state.

[1388] Device configuration and role

[1389] The terminal has the following functions:

[1390] 1. Uploading an electronic file: The user accesses the upload screen of the system, selects the electronic file they want to use as reference, and clicks the upload button.

[1391] 2. Entering a search prompt: The user enters an image or keyword about the material or product they want as a prompt.

[1392] 3. Emotion data collection: The emotion engine collects the user's voice data and facial image data and sends them to the server.

[1393] 4. Display search results: Display a customized list of suggested products and key information about each product to the user.

[1394] User operations

[1395] The user uses the system in the following steps:

[1396] 1. Uploading an electronic file: The user selects the electronic file they want to use as reference from the upload screen on their device and clicks the upload button. The device then sends the file to the server.

[1397] 2. Enter a search prompt: Enter an image or keyword of the material or product you want as a prompt. For example, enter "latest smartphone."

[1398] 3. Emotion recognition: The camera captures the user's facial image and voice, and the emotion engine analyzes the data to identify the user's emotional state. For example, it recognizes the emotion of "anxiety."

[1399] 4. Review search results: Review the customized list of suggested products sent from the server, and proceed to the detailed view or purchase process for the products that interest you.

[1400] Specific examples

[1401] For example, if a user types in "latest smartphone" and the emotion "impatient" is recognized from an image captured by the camera, the generated search prompt might look like this:

[1402] Prompt statement:

[1403] The latest smartphones with a feeling of anxiety

[1404] Based on this prompt, the server will provide quick and focused results, such as simpler, more readily available models and items that are in stock and ready to buy.

[1405] In this way, the system provides optimal products according to the user's emotional state and enables the user to quickly and efficiently obtain the information they are looking for.

[1406] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1407] Step 1:

[1408] The user accesses the system's upload screen, selects the electronic file they want to use as reference, and clicks the upload button. The terminal then sends this electronic file to the server. The input is the electronic file selected by the user, and the output is the transfer of the electronic file to the server.

[1409] Step 2:

[1410] The server stores the received electronic files and converts the file contents into text data using optical character recognition (OCR). The input is an electronic file, and the output is text data. This text data is organized into sections that separate each slide or page, and stored in a materials database.

[1411] Step 3:

[1412] The server analyzes the text data stored in the resource database using a natural language processing (NLP) engine, extracting key keywords and themes from titles, headings, and main text content. The input is text data, and the output is metadata (extracted keywords, themes, and importance information). This metadata is stored in the resource database.

[1413] Step 4:

[1414] The user enters a search prompt, and the terminal sends the prompt to the server. The input is the prompt sentence entered by the user, and the output is the transfer of the prompt sentence to the server.

[1415] Step 5:

[1416] The server analyzes the received search prompt and extracts relevant keywords. The input is the search prompt sentence and the output is the extracted keywords. The server then generates a search query based on these keywords.

[1417] Step 6:

[1418] The terminal transmits the user's voice data and facial image data to the emotion engine, which analyzes the data and recognizes the user's emotional state. The input is the voice data and facial image data, and the output is the recognized emotional state.

[1419] Step 7:

[1420] The server uses the generated search query to search a database of materials and pick out highly relevant materials and products. It also customizes search results based on the user's emotional state. For example, for a user who is feeling impatient, it prioritizes displaying short, focused materials and products. The input is the search query and the user's emotional state, and the output is customized search results.

[1421] Step 8:

[1422] The server sends customized proposal materials and product lists to the terminal. The terminal displays this information to the user. The input is the customized search results sent from the server, and the output is the display of the search results to the user. The user can view details of the materials or products they are interested in, download them, or proceed to purchase them.

[1423] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1424] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1425] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1426] [Fourth embodiment]

[1427] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1428] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1429] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1430] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1431] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1432] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1433] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1434] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1435] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1436] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1437] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1438] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1439] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1440] The present invention is a system for efficiently searching and suggesting reference materials, particularly those stored as electronic files, and automatically processing them to enable users to quickly and accurately obtain the information they desire.

[1441] This system is mainly composed of three entities: a server, a terminal, and a user. The specific operation of each is explained below.

[1442] Uploading and storing materials

[1443] User: First, the user accesses the system's upload screen, selects the electronic file (e.g., PDF, PPT, etc.) they want to use as reference, and clicks the upload button.

[1444] Terminal: Sends the file selected by the user to the server.

[1445] Server: The received file is stored in temporary storage. Then, using OCR (Optical Character Recognition) technology, the contents of the uploaded file are converted into text data. After conversion, each slide or page in the file is divided into sections and assigned a unique ID. This text data and section information are stored in a material database.

[1446] Text analysis and feature extraction of materials

[1447] Server: Analyzes text data stored in the resource database using a natural language processing (NLP) engine. Extracts key keywords and themes from titles, headings, and main text, and evaluates the importance of each section. The extracted keywords, themes, and importance information are stored in the resource database as metadata.

[1448] Entering and parsing search prompts

[1449] User: Enter the image or keywords of the desired material as a prompt.

[1450] Terminal: Sends the entered prompt to the server.

[1451] Server: Analyzes the received prompts, extracts relevant themes and keywords, and generates search queries based on the extracted themes and keywords.

[1452] Search and suggest related materials

[1453] Server: Searches the resource database using the generated search query. It compares the metadata in the resource database to select multiple highly relevant resources. It then extracts important slides and sections from each of the selected resources and generates summary information for each (title, keywords, importance, etc.). It then sends a list of the selected resources and key sections to the user's device.

[1454] Displaying the results

[1455] Terminal: Displays the user a list of proposal materials and the important slides and sections of each material. For each material, the title, summary, keywords, and thumbnails of important slides are displayed.

[1456] Users: Review the proposed materials and, if interested, view details or download them.

[1457] Specific examples

[1458] For example, consider the case where a user uploads a slide set for a conference presentation, "Research Presentation_Machine Learning.ppt." When the user selects a file using the upload screen and clicks the upload button, the device sends the file to the server. The server saves the received file in temporary storage and converts it into text data using OCR technology. The converted text data is divided into sections, and the information is stored in a materials database.

[1459] Next, when a user enters a prompt such as "latest applications of machine learning" into the search screen, the prompt is sent from the device to the server. The server analyzes the prompt and extracts keywords such as "machine learning," "latest," and "application." A search query is generated based on the extracted keywords, and by searching the document database, highly relevant documents are picked up.

[1460] Important slides (e.g., slides 3 and 7) are extracted from the selected materials, and summary information is generated for each. This information is sent to the user's device and displayed as a list of proposed materials.

[1461] Users can select the materials they want to know more about from the suggested list and view or download them, allowing them to obtain the information they are looking for quickly and efficiently.

[1462] Thus, the present invention is significantly more efficient than existing document retrieval systems and can quickly provide information that meets the user's needs.

[1463] The processing flow will be explained below.

[1464] Step 1: Upload your materials

[1465] User: The user accesses the system's upload screen, selects the electronic file (e.g., PDF, PPT, etc.) they want to use as reference, and clicks the upload button.

[1466] Step 2: Send the file

[1467] Terminal: Sends the file selected by the user to the server.

[1468] Step 3: Save the file

[1469] Server: Stores received files in temporary storage.

[1470] Step 4: OCR processing

[1471] Server: Launches an OCR (optical character recognition) engine and converts the contents of the uploaded file into text data.

[1472] Step 5: Sectioning

[1473] Server: Each slide or page in the text data file is divided into sections, and a unique ID is assigned to each section.

[1474] Step 6: Store in the database

[1475] Server: Stores the converted text data and section information in the material database.

[1476] Step 7: Natural Language Processing Analysis

[1477] Server: Analyzes the text data stored in the material database using a natural language processing (NLP) engine.

[1478] Step 8: Extracting keywords and themes

[1479] Server: Extracts key keywords and themes from titles, headings, and body content, and rates the importance of each section.

[1480] Step 9: Generate Metadata

[1481] Server: Generates extracted keywords, themes, and importance information as metadata and adds them to the material database.

[1482] Step 10: Enter the search prompt

[1483] User: Accesses the system's search screen and enters the image or keywords of the desired material as prompted.

[1484] Step 11: Sending prompts

[1485] Terminal: Sends the entered prompt to the server.

[1486] Step 12: Parse prompts

[1487] Server: Parses the received prompts and extracts relevant themes and keywords.

[1488] Step 13: Generate a search query

[1489] Server: Generates search queries based on the extracted themes and keywords.

[1490] Step 14: Search the documentation database

[1491] Server: Searches the resource database using the generated search query.

[1492] Step 15: Identify relevant materials

[1493] Server: Compares the metadata in the material database and selects multiple highly relevant materials.

[1494] Step 16: Extracting important sections

[1495] Server: Extracts key slides and sections from each document.

[1496] Step 17: Generate summary information

[1497] Server: Generates summary information (title, keywords, importance, etc.) for the extracted important slides and sections.

[1498] Step 18: Submit your proposal list

[1499] Server: Sends the list of proposal materials and the list of important sections to the user's terminal.

[1500] Step 19: View the results

[1501] Terminal: Displays the user a list of proposal materials and key slides and sections of each material.

[1502] Step 20: View details and download

[1503] Users: Review the suggested materials and select the ones that interest them for detailed viewing or download.

[1504] In this way, the system provides the user with the materials they require efficiently and quickly through a series of steps.

[1505] Example 1

[1506] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1507] Conventional reference material search systems have limitations in their ability to efficiently search and suggest the contents of electronic files, making it difficult for users to quickly and accurately obtain the information they need. Furthermore, they lacked the technology to improve the accuracy of suggesting highly relevant materials and the ability to extract and display important sections of materials. This resulted in the problem of users spending a great deal of time and effort finding the information they were looking for.

[1508] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1509] In this invention, the server includes: means for receiving electronic files of reference materials; means for converting the contents of the received electronic files into text data using optical character recognition technology; means for analyzing the text data with a natural language processing engine to extract keywords and themes and generate metadata; means for storing the metadata in a database; means for receiving search prompts and extracting relevant keywords from the prompts; means for searching the database based on the extracted keywords to select relevant materials; means for extracting important sections from the selected materials and providing them to the user; means for assigning unique IDs to the extracted sections and evaluating their importance; means for extracting themes from the prompts using a machine learning model; means for recommending highly relevant materials based on the metadata of the materials stored in the database; means for generating summary information of the relevant materials and transmitting it to the user terminal; and means for analyzing the search prompts and generating search queries. This allows users to quickly and efficiently obtain the information they need and effectively search for and suggest highly relevant materials.

[1510] "Electronic reference files" refers to materials and documents stored in digital formats such as PDF and PPT.

[1511] The "means for receiving" refers to a function for acquiring an electronic file when the server receives an uploaded file from a user.

[1512] "Optical character recognition technology" is a technology called OCR (Optical Character Recognition), which analyzes characters in an image and converts them into text data.

[1513] "Text data" refers to data in which only character information is extracted from the contents of an electronic file and saved in text format.

[1514] A "natural language processing engine" is a computer science technology for analyzing text data to extract keywords and themes.

[1515] "Metadata" refers to data that includes keywords, themes, section importance information, and the like extracted from text data.

[1516] A "database" is a system for efficiently storing, retrieving, and managing various types of data.

[1517] A "search prompt" is a keyword or phrase that a user enters to search for the information they need.

[1518] "Means for extracting relevant keywords" refers to a function for extracting important keywords from a search prompt.

[1519] A "search query" is a specific search condition generated to search a database.

[1520] "Means for picking out relevant materials" refers to a function for selecting relevant materials from a database based on a search query.

[1521] "Means for extracting important sections" refers to a function for selecting particularly important parts from the picked-up material.

[1522] A "unique ID" is an identifier assigned to each section to uniquely identify it.

[1523] The "means for evaluating the importance" refers to a function for quantitatively evaluating the importance of the extracted section.

[1524] A "machine learning model" is an algorithm or technique that learns from data and makes predictions or classifications based on that data.

[1525] "Means for recommending highly relevant materials" refers to a function for selecting and suggesting materials that best fit the user's search prompt.

[1526] The "means for generating summary information and transmitting it to the user terminal" refers to a function for generating a summary of the picked-up material and transmitting it to the user terminal.

[1527] The present invention is a system for efficiently searching and suggesting reference materials. This system automatically processes materials stored as electronic files, enabling users to quickly and accurately obtain the information they desire. The system is primarily composed of three entities: a server, a terminal, and a user. The specific operation of each is described below.

[1528] Uploading and storing materials

[1529] Users access the system's upload screen, select the electronic file they want to use as reference (e.g. PDF, PPT, etc.), and click the upload button. The terminal sends the file selected by the user to the server. The server stores the received file in temporary storage. The server then uses OCR (optical character recognition) technology to convert the contents of the uploaded file into text data. Tesseract OCR is used for the conversion. After conversion, each slide or page in the file is divided into sections and assigned a unique ID. This text data and section information are stored in a material database. The material database uses MySQL.

[1530] Text analysis and feature extraction of materials

[1531] The server analyzes the text data stored in the document database using a natural language processing (NLP) engine. The NLP engine used is SpaCy. It extracts key keywords and themes from titles, headings, and body content, and uses the TF-IDF algorithm to evaluate the importance of each section. The extracted keywords, themes, and importance information are stored in the document database as metadata.

[1532] Entering and parsing search prompts

[1533] The user enters an image of the desired material or keywords as a prompt into the search screen. The device sends the entered prompt to the server. The server analyzes the received prompt and extracts related themes and keywords. The BERT model is used to analyze the prompt. A search query is generated based on the extracted keywords.

[1534] Search and suggest related materials

[1535] The server searches the document database using the generated search query. It compares the results with the metadata in the document database to select multiple highly relevant documents. It calculates a relevance score and selects the top documents. It extracts the most important slides and sections from each of the selected documents and generates summary information (title, keywords, importance, etc.) for each. The generated summary information is sent from the server to the user's device in JSON format.

[1536] Displaying the results

[1537] The terminal displays a list of proposed materials and the important slides and sections of each material to the user. For each material, the title, summary, keywords, and thumbnails of important slides are displayed. HTML / CSS and JavaScript are used to display the user interface. The user can check the proposed materials and, if interested, can view details or download them. Downloads are performed using an HTTP GET request.

[1538] Specific examples

[1539] For example, consider the case where a user uploads a slide set for a conference presentation, "Research Presentation_Machine Learning.ppt." When the user selects a file using the upload screen and clicks the upload button, the device sends the file to the server. The server saves the received file in temporary storage and converts it into text data using OCR technology. The converted text data is divided into sections, and the information is stored in a materials database.

[1540] Next, if a user enters a prompt such as "latest applications of machine learning" into the search screen, the prompt is sent from the device to the server. The server analyzes the prompt and extracts keywords such as "machine learning," "latest," and "application." A search query is generated based on the extracted keywords, and highly relevant materials are picked out by searching the materials database. Important slides (e.g., slides 3 and 7) are extracted from the picked materials, and summary information is generated for each. This information is sent to the user's device and displayed as a list of suggested materials.

[1541] This system allows users to quickly and efficiently obtain the information they need, significantly improving the efficiency of research and work. As a result, it becomes possible to provide information that meets the user's needs appropriately.

[1542] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1543] Step 1:

[1544] The user accesses the upload screen of the system, selects the electronic file they want to use as reference, and clicks the upload button.

[1545] Input: An electronic file selected by the user (e.g. PDF, PPT, etc.)

[1546] Output: Request to upload selected files

[1547] Step 2:

[1548] The terminal transmits the file selected by the user to the server.

[1549] Input: User upload request and selected electronic file

[1550] Output: A request for an electronic file to be sent to the server

[1551] Step 3:

[1552] The server stores the received file in temporary storage.

[1553] Input: Electronic file sent from the terminal

[1554] Output: Electronic file saved in temporary storage

[1555] Step 4:

[1556] The server uses OCR technology to convert the contents of the uploaded file into text data.

[1557] Input: Electronic files stored in temporary storage

[1558] Output: Converted text data

[1559] Step 5:

[1560] The server divides the text data into sections and assigns each a unique ID.

[1561] Input: Converted text data

[1562] Output: Text data with a unique ID assigned to each section

[1563] Step 6:

[1564] The server stores the section information and text data in a material database.

[1565] Input: Text data divided into sections

[1566] Output: Text data stored in the material database and its section information

[1567] Step 7:

[1568] The server uses a natural language processing engine to analyze the text data stored in the document database, extract key keywords and themes, and evaluate the importance of each section.

[1569] Input: Text data stored in the material database

[1570] Output: Extracted keywords, themes, and importance of each section

[1571] Step 8:

[1572] The server stores the extracted keywords, themes, and importance information as metadata in the document database.

[1573] Input: Extracted keywords, themes, and importance information

[1574] Output: Metadata stored in the material database

[1575] Step 9:

[1576] The user enters an image of the material they want or keywords into the search screen as a prompt.

[1577] Input: Search prompt

[1578] Output: User search request

[1579] Step 10:

[1580] The terminal sends the entered prompt to the server.

[1581] Input: User search request

[1582] Output: Sending a search prompt to the server

[1583] Step 11:

[1584] The server analyzes the prompts it receives and uses machine learning models to extract relevant themes and keywords.

[1585] Input: Submitted search prompt

[1586] Output: Themes and keywords extracted from the prompt

[1587] Step 12:

[1588] The server generates a search query based on the extracted themes and keywords.

[1589] Input: Extracted themes and keywords

[1590] Output: Generated search query

[1591] Step 13:

[1592] The server searches the document database using the generated search query and picks out multiple highly relevant documents.

[1593] Input: Generated search query

[1594] Output: Related materials picked up

[1595] Step 14:

[1596] The server extracts the most important slides and sections from each document and generates a summary.

[1597] Input: Picked up materials

[1598] Output: Summary information (title, keywords, importance, etc.)

[1599] Step 15:

[1600] The server transmits a list of proposal materials including summary information to the user's terminal.

[1601] Input: Summary Information

[1602] Output: A list of proposal materials sent to the user's device

[1603] Step 16:

[1604] The device displays a list of proposal materials and important slides and sections of each material to the user.

[1605] Input: Proposal list sent from the server

[1606] Output: A list of proposals displayed to the user

[1607] Step 17:

[1608] The user can review the proposed materials and, if interested, view them in detail or download them.

[1609] Input: Displayed proposal list

[1610] Output: View detailed view or download document

[1611] Through the above steps, the user can quickly and efficiently obtain the information he or she needs.

[1612] (Application example 1)

[1613] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1614] Factories and industrial sites often contain vast amounts of technical documents and manuals, but it is not easy to quickly and accurately search through them and obtain the information needed. Managing and searching through these documents takes a great deal of time and effort, especially when there are many paper documents and printed materials. This situation reduces work efficiency and impacts productivity. Furthermore, delays in accessing necessary information can lead to delays in equipment maintenance and troubleshooting. To address these challenges, an effective document management and search system is needed.

[1615] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1616] In this invention, the server includes: means for receiving electronic files of reference materials; means for converting the contents of the received electronic files into text data using optical character recognition technology; means for analyzing the text data with a natural language processing engine to extract keywords and themes and generate metadata; means for storing a database for storing the metadata; means for receiving search prompts and extracting relevant keywords from the prompts; means for searching a database based on the extracted keywords to find relevant materials; means for extracting important sections from the found materials and providing them to a user; means for a user to scan paper documents or printed materials with a smartphone and upload the images; means for processing the uploaded images with OCR technology and converting them to text; and means for searching a database using a TF-IDF-based search algorithm to suggest relevant materials in response to a user's search query, enabling factories and industrial sites to quickly and accurately search for necessary materials and provide them to users.

[1617] "Electronic files of reference materials" are technical documents and manuals related to work in factories and industrial sites that are stored in digital format.

[1618] Optical character recognition (OCR) is a technology that scans characters written on paper documents or printed materials and converts them into digital character data.

[1619] "Text data" refers to the text information in electronic files converted using OCR technology and stored in digital format.

[1620] A "natural language processing engine (NLP)" is a program that analyzes text data and automatically extracts important keywords and themes.

[1621] "Metadata" refers to additional information extracted from text data, such as keywords, themes, and importance, which is used to efficiently search and organize data within a database.

[1622] A "database" is a system for centrally managing and storing metadata and related material information.

[1623] A "search prompt" is a keyword or query that a user enters to specifically describe the information they are seeking.

[1624] A "TF-IDF-based search algorithm" is a statistical method for evaluating the importance of documents in a database based on a search query and deriving highly relevant materials.

[1625] "Scanning with a smartphone" refers to the process by which a user uses the smartphone's camera function to convert paper documents or printed materials into digital images.

[1626] "Uploading" is the act of a user transferring a digital image or electronic file to a cloud system such as a server.

[1627] "Important sections" are those parts of the selected material that are particularly relevant and useful to the user.

[1628] The present invention is a document management and search system for use in factories and industrial sites. This system is composed of three main components: a server, a terminal, and a user. The specific operation of each component will be explained below.

[1629] Uploading and storing materials

[1630] User: The user uses their smartphone to scan paper documents or printed materials and convert them into digital images, then selects the scanned image on the application's upload screen and clicks the upload button.

[1631] Terminal: Sends the image file selected by the user to the server.

[1632] Server: The received image files are stored in temporary storage and the contents are converted into text data using OCR (Optical Character Recognition) technology. After conversion, each file is divided into sections and assigned a unique ID. This text data and section information are stored in a material database.

[1633] Text analysis and feature extraction of materials

[1634] Server: Analyzes text data stored in the resource database using a natural language processing (NLP) engine. Extracts key keywords and themes and evaluates the importance of each section. The extracted keywords, themes, and importance information are stored in the resource database as metadata.

[1635] Entering and parsing search prompts

[1636] User: The user enters a search prompt into the application for the required material, for example, "Basic steps in PLC programming."

[1637] Terminal: Sends the entered prompt to the server.

[1638] Server: Analyzes the received prompts, extracts relevant themes and keywords, and generates search queries based on the extracted themes and keywords.

[1639] Search and suggest related materials

[1640] Server: Searches the document database using the generated search query. It uses a TF-IDF-based search algorithm to evaluate the importance of documents in the database and selects highly relevant documents. It then extracts important slides and sections from each of the selected documents and generates summary information for each (title, keywords, importance, etc.). It then sends a list of the selected documents and key sections to the user's device.

[1641] Displaying the results

[1642] Terminal: Displays the user a list of proposed materials and the important sections of each material. For each material, displays the title, summary, keywords, and thumbnails of important sections.

[1643] Users: Review the proposed materials and, if interested, view details or download them.

[1644] The system's hardware mainly requires a smartphone with a camera and a high-performance server. It uses the pytesseract library for OCR technology, and the nltk and scikit-learn libraries for natural language processing. The TF-IDF algorithm is used for data analysis of the documents.

[1645] Specific examples

[1646] For example, consider a user on a factory floor looking for a "Troubleshooting Guide for PLC Programming." The user scans a paper manual with their smartphone and uploads it to the server using an application. Next, they input "Basic Steps for PLC Programming" as a search prompt. The server analyzes this prompt and uses a TF-IDF-based algorithm to search and suggest relevant documents from the database. The user then checks the detailed view of the displayed document list to obtain the information they need.

[1647] The above method makes it possible to quickly and accurately search for and provide users with the necessary documents and information at factories and industrial sites. This system not only improves work efficiency at the site, but also speeds up troubleshooting and maintenance work by providing timely and appropriate information.

[1648] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1649] Step 1:

[1650] A user uses a smartphone to scan paper documents or printed materials and convert them into digital images. Then, they select the scanned image on the application's upload screen and click the upload button. At this point, the input is the scanned image file, and the output is an upload request.

[1651] Step 2:

[1652] The terminal sends the image file selected by the user to the server. The input here is the scanned image file, and the output is the image file sent to the server. This sending process is performed via the Internet.

[1653] Step 3:

[1654] The server stores the received image file in temporary storage. It then uses OCR technology to convert the image file into text data. The input here is the sent image file, and the output is text data. Specifically, it uses the pytesseract library to recognize the text in the image and convert it into text.

[1655] Step 4:

[1656] The server divides the converted text data into sections, assigns unique IDs, and stores them in a database. In this step, the input is text data, and the output is text data in a database divided into sections.

[1657] Step 5:

[1658] The server uses a natural language processing engine (NLP) to analyze the text data stored in the resource database and extract key keywords and themes. The input is the text data in the database, and the output is metadata. Specifically, it uses NLP technology to extract important words and phrases from the text and stores them as metadata.

[1659] Step 6:

[1660] The user enters a search prompt for the required material on the application search screen, for example, a keyword such as "Basic steps of PLC programming." The input of this step is the search prompt, and the output is a prompt transmission request from the terminal to the server.

[1661] Step 7:

[1662] The terminal sends the entered prompt to the server, where the input is the search prompt entered by the user and the output is a prompt transmission request to the server.

[1663] Step 8:

[1664] The server analyzes the prompts it receives, extracts relevant themes and keywords, and generates a search query based on that information. The input is the prompt, and the output is the generated search query. This analysis is performed using a generative AI model.

[1665] Step 9:

[1666] The server uses the generated search query to search the resource database, where the input is the search query and the output is a list of relevant resources. A TF-IDF-based search algorithm is used to evaluate and select relevant documents.

[1667] Step 10:

[1668] The server extracts important sections from each of the documents picked up and generates summary information (title, keywords, importance, etc.) for each. The input is a list of documents, and the output is a list of proposed documents with summary information.

[1669] Step 11:

[1670] The server sends the list of proposal materials and information on important sections of each material to the user's terminal. The input is the list of proposal materials with summary information, and the output is a transmission request to the user's terminal.

[1671] Step 12:

[1672] The terminal displays the list of proposal materials and important sections of each material to the user. The input is the list of proposal materials sent from the server, and the output is the material information displayed to the user.

[1673] Step 13:

[1674] The user checks the proposed materials and, if interested, can view details or download them. The input is the material information displayed to the user, and the output is the downloaded detailed materials.

[1675] This series of processes makes it possible to quickly and accurately search for necessary materials and information at factories and industrial sites and provide them to users.

[1676] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1677] The present invention is a system for efficiently searching electronic files of reference materials and quickly providing users with the information they desire. Furthermore, the present invention has the ability to recognize a user's emotions and customize search results based on those emotions. This system is specifically designed to automatically process materials stored as electronic files, enabling users to quickly and accurately obtain the information they desire.

[1678] This system is mainly composed of three entities: a server, a terminal, and a user. The specific operation of each is explained below.

[1679] Uploading and storing materials

[1680] User: The user accesses the system's upload screen, selects the electronic file (e.g., PDF, PPT, etc.) they want to use as reference, and clicks the upload button.

[1681] Terminal: Sends the file selected by the user to the server.

[1682] Server: The received file is stored in temporary storage. Then, using OCR (Optical Character Recognition) technology, the contents of the uploaded file are converted into text data. After conversion, each slide or page in the file is divided into sections and assigned a unique ID. This text data and section information are stored in a material database.

[1683] Text analysis and feature extraction of materials

[1684] Server: Analyzes text data stored in the resource database using a natural language processing (NLP) engine. Extracts key keywords and themes from titles, headings, and main text, and evaluates the importance of each section. The extracted keywords, themes, and importance information are stored in the resource database as metadata.

[1685] Emotion recognition by emotion engine

[1686] On the device: When a user searches, the emotion engine collects the user's voice data or facial image data.

[1687] Server: The emotion engine analyzes the collected data and recognizes the user's emotional state, for example, identifying whether the user is anxious or calm.

[1688] Entering and parsing search prompts

[1689] User: Enter the image or keywords of the desired material as a prompt.

[1690] Terminal: Sends the entered prompt to the server.

[1691] Server: Analyzes the received prompts, extracts relevant themes and keywords, and generates search queries based on the extracted themes and keywords.

[1692] Search and suggest related materials

[1693] Server: Searches the resource database using the generated search query, compares it with the resource database metadata, and selects multiple highly relevant resources.

[1694] Server: Extracts important slides and sections from each presentation and customizes the order and content of the presentation based on the user's perceived emotions. For example, prioritize short, concise presentations for impatient users.

[1695] Displaying the results

[1696] Terminal: Presents users with a customized list of presentation materials and key slides and sections for each material. For each presentation, it displays the title, summary, keywords, and thumbnails of key slides.

[1697] Users: Review the proposed materials and, if interested, view details or download them.

[1698] Specific examples

[1699] For example, consider the case where a user uploads a slide set for a conference presentation, "Research Presentation_Machine Learning.ppt." When the user selects a file using the upload screen and clicks the upload button, the device sends the file to the server. The server saves the received file in temporary storage and converts it into text data using OCR technology. The converted text data is divided into sections, and the information is stored in a materials database.

[1700] Next, when a user enters a prompt such as "latest applications of machine learning" into the search screen, the prompt is sent from the device to the server. The server analyzes the prompt and extracts keywords such as "machine learning," "latest," and "application." A search query is generated based on the extracted keywords, and by searching the document database, highly relevant documents are picked up.

[1701] At that time, the device sends the user's voice data and facial image data to the emotion engine, which then recognizes the user's emotions.The server then customizes the presentation order and content of the selected materials based on that emotional information.For example, if the user is feeling impatient, it will prioritize displaying materials that summarize only the main points.

[1702] Finally, the list of suggested materials is sent to the terminal, where the user can review the list and view or download the materials they are interested in. In this way, the system provides the most appropriate materials according to the user's emotional state, enabling the user to quickly and efficiently obtain the information they are looking for.

[1703] The processing flow will be explained below.

[1704] Step 1: Upload your materials

[1705] User: The user accesses the system's upload screen, selects the electronic file (e.g., PDF, PPT, etc.) they want to use as reference, and clicks the upload button.

[1706] Step 2: Send the file

[1707] Terminal: Sends the file selected by the user to the server.

[1708] Step 3: Save the file

[1709] Server: Stores received files in temporary storage.

[1710] Step 4: OCR processing

[1711] Server: Launches an OCR (optical character recognition) engine and converts the contents of the uploaded file into text data.

[1712] Step 5: Sectioning

[1713] Server: Each slide or page in the text data file is divided into sections, and a unique ID is assigned to each section.

[1714] Step 6: Store in the database

[1715] Server: Stores the converted text data and section information in the material database.

[1716] Step 7: Natural Language Processing Analysis

[1717] Server: Analyzes the text data stored in the material database using a natural language processing (NLP) engine.

[1718] Step 8: Extracting keywords and themes

[1719] Server: Extracts key keywords and themes from titles, headings, and body content, and rates the importance of each section.

[1720] Step 9: Generate Metadata

[1721] Server: Generates extracted keywords, themes, and importance information as metadata and adds them to the material database.

[1722] Step 10: Enter the search prompt

[1723] User: Accesses the system's search screen and enters the image or keywords of the desired material as prompted.

[1724] Step 11: Sending prompts

[1725] Terminal: Sends the entered prompt to the server.

[1726] Step 12: Parse prompts

[1727] Server: Parses the received prompts and extracts relevant themes and keywords.

[1728] Step 13: Generate a search query

[1729] Server: Generates search queries based on the extracted themes and keywords.

[1730] Step 14: Collect emotion data

[1731] Device: In parallel with entering the search prompt, the user's voice data and facial image data are sent to the emotion engine.

[1732] Step 15: Analyze the emotion data

[1733] Server: The emotion engine analyzes the collected voice data and facial image data to recognize the user's emotional state, for example, determining whether they are "anxious" or "calm."

[1734] Step 16: Search the documentation database

[1735] Server: Searches the resource database using the generated search query.

[1736] Step 17: Identify relevant materials

[1737] Server: Compares the metadata in the material database and selects multiple highly relevant materials.

[1738] Step 18: Extracting important sections

[1739] Server: Extracts key slides and sections from each document.

[1740] Step 19: Emotional Customization

[1741] Server: Customize the presentation order and content of the selected materials based on the user's recognized emotional state. For example, prioritize displaying materials that summarize the main points to a user who is feeling impatient.

[1742] Step 20: Submit your proposal list

[1743] Server: Sends a customized list of proposal materials and a list of important sections to the user's device.

[1744] Step 21: View the results

[1745] Terminal: Displays the user a list of proposal materials and key slides and sections of each material.

[1746] Step 22: View details and download

[1747] Users: Review the suggested materials and select those that interest them for detailed viewing or download.

[1748] Example 2

[1749] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1750] Conventional document search systems have difficulty in quickly and accurately providing the information users are looking for. Furthermore, because they provide uniform search results without considering the user's emotions or state, it is difficult to provide information that meets the user's needs. Furthermore, there are often too many search results, which can lead to important information being overlooked, resulting in a decrease in user satisfaction.

[1751] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1752] In this invention, the server includes means for receiving electronic files of reference materials, means for converting the contents of the received electronic files into text data using optical character recognition technology, means for analyzing the text data with a natural language processing engine to extract keywords and themes and generate metadata, means for analyzing audio data or image data and recognizing the user's emotions, means for searching a database based on the extracted keywords and the recognized emotions to select relevant materials, and means for extracting important sections from the selected materials and providing them to the user. This makes it possible to provide appropriate information according to the user's emotional state, enabling the user to quickly and accurately obtain the information they are looking for.

[1753] "Reference materials" refers to electronic files that users use to obtain information, such as PDFs and PPTs.

[1754] "Optical character recognition technology" is a technology that reads characters from image data and converts them into text data.

[1755] "Text data" refers to data that contains only text information and that has been converted using optical character recognition technology.

[1756] "Natural language processing engine" is a general term for software and algorithms that analyze text data and extract keywords and themes.

[1757] "Metadata" refers to data attribute information, such as keywords, themes, and relevance evaluation information extracted from text data.

[1758] A "search prompt" refers to the keywords or phrases that a user enters about the search target.

[1759] "Emotion recognition" is a technology that analyzes voice data and image data to identify a user's emotional state.

[1760] "Database" refers to an information system that stores data in a structured way to facilitate retrieval and access.

[1761] "Key sections" are those parts of the document that are assessed as being particularly important and include key points and summary information.

[1762] A "thumbnail" is a small version of a document or image that allows you to easily check the content.

[1763] A "machine learning model" refers to an algorithm or model that learns patterns based on data and makes predictions and classifications.

[1764] The present invention is a system for efficiently searching electronic files of reference materials and quickly providing users with the information they desire. Furthermore, the present invention has the ability to recognize users' emotions and customize search results based on those emotions. This system is specifically designed to automatically process materials stored as electronic files, enabling users to quickly and accurately obtain the information they desire.

[1765] This system is mainly composed of three entities: a server, a terminal, and a user. The specific operations performed by each entity are shown below.

[1766] Server Functions and Operations

[1767] The server performs processing using the following hardware and software.

[1768] Hardware: High-performance processor, memory, and storage

[1769] Software: OCR technology (e.g., Tesseract), natural language processing engines (e.g., spaCy, NLTK), emotion recognition engines (e.g., Microsoft Azure Cognitive Services, OpenCV)

[1770] The server first receives the electronic file uploaded by the user and stores it in temporary storage. Next, it uses OCR technology to convert the contents of the electronic file into text data. After conversion, each slide or page in the file is divided into sections and assigned a unique ID. This text data and section information are then stored in a material database.

[1771] The text data stored in the resource database is analyzed using a natural language processing engine. Key keywords and themes are extracted from the title, headings, and body of the text, and the importance of each section is evaluated. The extracted keywords, themes, and importance information are saved in the resource database as metadata.

[1772] The emotion engine analyzes the voice data or facial image data collected when the user searches to recognize the user's emotional state, which is also stored in the database.

[1773] The server analyzes the search prompt sent by the user and extracts relevant themes and keywords using a generative AI model (e.g., BERT, GPT-3). It then generates a search query based on the extracted themes and keywords and searches the document database.

[1774] The generated search query is used to search the document database to select multiple highly relevant documents, extract important slides and sections from each document, and customize the order and content of the documents to be presented based on the user's perceived emotions.

[1775] Device features and operations

[1776] A terminal is a device that a user uses as an interface, and performs processing using the following hardware and software.

[1777] Hardware: microphone, camera, display

[1778] Software: Web browser, communication module

[1779] When a user uploads a document, the terminal transmits the electronic file selected by the user to the server. When a user enters a search prompt, the terminal also collects the user's voice data and facial image data and transmits them to the server.

[1780] User operations

[1781] Users access the system's upload screen, select the electronic file they want to use as reference, and click the upload button. They also enter keywords related to the information they are looking for as prompted on the search screen. They can check the list of suggested materials displayed on the terminal and view or download the materials they are interested in in detail.

[1782] Specific examples

[1783] For example, consider the case where a user uploads a slide set for a conference presentation, "Research Presentation_Machine Learning.ppt." When the user selects a file using the upload screen and clicks the upload button, the device sends the file to the server. The server saves the received file in temporary storage and converts it into text data using OCR technology. The converted text data is divided into sections, and the information is stored in a materials database.

[1784] Next, when a user enters a prompt such as "latest applications of machine learning" into the search screen, the prompt is sent from the device to the server. The server analyzes the prompt and extracts keywords such as "machine learning," "latest," and "application." A search query is generated based on the extracted keywords, and by searching the document database, highly relevant documents are picked up.

[1785] At that time, the device sends the user's voice data and facial image data to the emotion engine, which then recognizes the user's emotions.The server then customizes the presentation order and content of the selected materials based on that emotional information.For example, if the user is feeling impatient, it will prioritize displaying materials that summarize only the main points.

[1786] Finally, the list of suggested materials is sent to the terminal, and the user can check the list and view or download the materials of interest in detail.

[1787] Examples of prompt statements

[1788] The prompt to be input to the generative AI model is as follows:

[1789] "I'm looking for the latest papers related to my research topic."

[1790] "What's the latest news on automotive battery technology?"

[1791] "I'd like to know what to do about this week's economic trends."

[1792] In this way, the system allows users to quickly and efficiently obtain the information they desire.

[1793] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1794] Step 1: Upload and store your materials

[1795] Input: An electronic file selected by the user (e.g. PDF, PPT, etc.)

[1796] Output: Text data and section information are stored in a resource database.

[1797] Specific behavior:

[1798] User: The user accesses the system's upload screen, selects the electronic file they wish to upload, and clicks the upload button.

[1799] Terminal: Sends the file selected by the user to the server. The terminal encodes the file as binary data and sends it via an HTTP request.

[1800] Server: The server stores the received files in temporary storage. It then uses Tesseract OCR technology to convert the file contents into text data. The converted text data is divided into sections for each slide or page, and a unique ID is assigned. This text data and section information are stored in a material database.

[1801] Step 2: Text analysis and feature extraction of the material

[1802] Input: Text data stored in the resource database

[1803] Output: Generates metadata for keywords, themes, and importance information

[1804] Specific behavior:

[1805] Server: The server analyzes the text data stored in the document database using a natural language processing (NLP) engine. Software used includes spaCy and the NLTK library. After analysis, it extracts key keywords and themes from the title, headings, and body of the text. For example, for a document containing the theme of "machine learning," it extracts keywords such as "neural network" and "regression analysis." It then evaluates the importance of the extracted keywords and themes, as well as each section, and stores this information as metadata in the document database.

[1806] Step 3: Emotion recognition by the emotion engine

[1807] Input: User's voice or facial image data

[1808] Output: User's emotional information

[1809] Specific behavior:

[1810] Device: When a user searches, the device collects the user's voice data or facial image data. This is done using a microphone or camera, which records and photographs the data in real time.

[1811] Server: The emotion engine analyzes the collected voice data and facial image data to recognize the user's emotional state. For voice data, it uses the emotion recognition API from Microsoft Azure Cognitive Services, and for facial image data, it uses OpenCV and dlib algorithms to identify whether the user is anxious or calm. This emotional information is stored in a database.

[1812] Step 4: Entering and Parsing Search Prompts

[1813] Input: The search prompt entered by the user

[1814] Output: The generated search query

[1815] Specific behavior:

[1816] User: The user enters a keyword or phrase related to the information they are looking for as a prompt in the search screen. For example, they might enter "latest applications of machine learning."

[1817] Terminal: Sends the entered prompt to the server. The prompt is encoded as string data and sent through an HTTP request.

[1818] Server: The server analyzes the received prompt using a natural language processing model (e.g., GPT-3) to extract relevant themes and keywords, such as "machine learning," "latest," and "applied," and generates a search query based on these.

[1819] Step 5: Search and suggest related materials

[1820] Input: Generated search query and sentiment information

[1821] Output: Related materials picked up

[1822] Specific behavior:

[1823] Server: Using the generated search query, the server searches the resource database. It uses information retrieval algorithms such as TF-IDF and Doc2Vec to match the resource database metadata and find relevant resources.

[1824] Server: Extracts important slides and sections from each presentation. It also customizes the order and content of presentation materials based on the user's perceived emotions. For example, if the user is feeling impatient, it prioritizes short, focused presentation materials.

[1825] Step 6: View the results

[1826] Input: Customized proposal list

[1827] Output: Display the list of materials to the user

[1828] Specific behavior:

[1829] Terminal: The terminal displays the customized list of proposal materials sent from the server to the user. For each material, the title, summary, keywords, and thumbnails of important slides are displayed in a user interface using HTML and CSS.

[1830] User: The user can view the list of proposal documents and find the ones they are interested in for detailed view or download. For example, they can click on the title of a proposal document to access its detailed page and download it in PDF format.

[1831] The above is the flow of program processing for this system.

[1832] (Application example 2)

[1833] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1834] In today's information-saturated society, it is difficult for users to quickly and accurately obtain the specific information they are looking for. In particular, there is no method to efficiently search through materials stored in electronic files and provide appropriate information based on the user's emotions and situation. Furthermore, online shopping sites are in need of a system that can perform personalized product searches that take the user's emotions into account.

[1835] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1836] In this invention, the server includes means for receiving electronic files of reference materials, means for converting the contents of the received electronic files into text data using optical character recognition technology, means for analyzing the text data using a natural language processing engine to extract keywords and themes and generate metadata, and means for recognizing the user's emotions and customizing search results based on the user's emotional state. This enables the user to quickly and accurately obtain the information they are looking for, and further enables the provision of personalized information according to the user's emotions.

[1837] "Electronic reference files" are digital files that store information uploaded by users to the system, including formats such as PDF and PPT.

[1838] "Optical character recognition technology" is a technology that extracts text data from image data, and is also known as OCR (Optical Character Recognition).

[1839] "Text data" refers to digital data that contains textual information and has been converted into a format that can be used for analysis and searching.

[1840] A "natural language processing engine" is a computer program for understanding and analyzing human language, and is used to extract keywords and themes from text data.

[1841] "Metadata" is data that contains information about data, such as keywords, themes, and importance of text data stored in a material database.

[1842] A "database" is a collection of data that is stored and managed electronically and that can be searched and retrieved by users.

[1843] A "search prompt" is a keyword or phrase that a user uses to input the information they are seeking and is used in the system's search function.

[1844] "Emotional state" refers to the user's current emotional state, which the system recognizes by analyzing voice data and image data.

[1845] "Personalization" refers to adapting and customizing information based on a user's individual characteristics and feelings.

[1846] "Important sections" refer to parts or slides that are considered to be particularly important in the materials, and are extracted to quickly provide users with the information they need.

[1847] This invention is a system that recognizes a user's emotions and customizes search results based on a specific emotional state. This system is particularly applicable to online shopping sites, where users not only input images and keywords of the items they want, but also recognize emotions from facial images and voice, and suggest products that best suit those emotions. This system is primarily composed of three entities: a server, a terminal, and the user.

[1848] Server configuration and roles

[1849] The server has the following functions:

[1850] 1. Receiving and processing electronic files: Electronic files uploaded by users are received and converted into text data using OCR technology. This text data is then analyzed using a natural language processing engine to extract keywords and themes, generating metadata, and storing it in the material database.

[1851] 2. Emotion recognition: It has the function of recognizing the user's emotions, and analyzes the voice data and facial image data collected by the emotion engine to identify the user's emotional state.

[1852] 3. Search prompt analysis and customization: Analyze the search prompt received from the user and extract relevant keywords, then customize the search results based on the user's emotional state.

[1853] 4. Product Search and Recommendation: The generated search query is used to search the document database to find relevant products, and then personalized product recommendations are made based on the user's emotional state.

[1854] Device configuration and role

[1855] The terminal has the following functions:

[1856] 1. Uploading an electronic file: The user accesses the upload screen of the system, selects the electronic file they want to use as reference, and clicks the upload button.

[1857] 2. Entering a search prompt: The user enters an image or keyword about the material or product they want as a prompt.

[1858] 3. Emotion data collection: The emotion engine collects the user's voice data and facial image data and sends them to the server.

[1859] 4. Display search results: Display a customized list of suggested products and key information about each product to the user.

[1860] User operations

[1861] The user uses the system in the following steps:

[1862] 1. Uploading an electronic file: The user selects the electronic file they want to use as reference from the upload screen on their device and clicks the upload button. The device then sends the file to the server.

[1863] 2. Enter a search prompt: Enter an image or keyword of the material or product you want as a prompt. For example, enter "latest smartphone."

[1864] 3. Emotion recognition: The camera captures the user's facial image and voice, and the emotion engine analyzes the data to identify the user's emotional state. For example, it recognizes the emotion of "anxiety."

[1865] 4. Review search results: Review the customized list of suggested products sent from the server, and proceed to the detailed view or purchase process for the products that interest you.

[1866] Specific examples

[1867] For example, if a user types in "latest smartphone" and the emotion "impatient" is recognized from an image captured by the camera, the generated search prompt might look like this:

[1868] Prompt statement:

[1869] The latest smartphones with a feeling of anxiety

[1870] Based on this prompt, the server will provide quick and focused results, such as simpler, more readily available models and items that are in stock and ready to buy.

[1871] In this way, the system provides optimal products according to the user's emotional state and enables the user to quickly and efficiently obtain the information they are looking for.

[1872] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1873] Step 1:

[1874] The user accesses the system's upload screen, selects the electronic file they want to use as reference, and clicks the upload button. The terminal then sends this electronic file to the server. The input is the electronic file selected by the user, and the output is the transfer of the electronic file to the server.

[1875] Step 2:

[1876] The server stores the received electronic files and converts the file contents into text data using optical character recognition (OCR). The input is an electronic file, and the output is text data. This text data is organized into sections that separate each slide or page, and stored in a materials database.

[1877] Step 3:

[1878] The server analyzes the text data stored in the resource database using a natural language processing (NLP) engine, extracting key keywords and themes from titles, headings, and main text content. The input is text data, and the output is metadata (extracted keywords, themes, and importance information). This metadata is stored in the resource database.

[1879] Step 4:

[1880] The user enters a search prompt, and the terminal sends the prompt to the server. The input is the prompt sentence entered by the user, and the output is the transfer of the prompt sentence to the server.

[1881] Step 5:

[1882] The server analyzes the received search prompt and extracts relevant keywords. The input is the search prompt sentence and the output is the extracted keywords. The server then generates a search query based on these keywords.

[1883] Step 6:

[1884] The terminal transmits the user's voice data and facial image data to the emotion engine, which analyzes the data and recognizes the user's emotional state. The input is the voice data and facial image data, and the output is the recognized emotional state.

[1885] Step 7:

[1886] The server uses the generated search query to search a database of materials and pick out highly relevant materials and products. It also customizes search results based on the user's emotional state. For example, for a user who is feeling impatient, it prioritizes displaying short, focused materials and products. The input is the search query and the user's emotional state, and the output is customized search results.

[1887] Step 8:

[1888] The server sends customized proposal materials and product lists to the terminal. The terminal displays this information to the user. The input is the customized search results sent from the server, and the output is the display of the search results to the user. The user can view details of the materials or products they are interested in, download them, or proceed to purchase them.

[1889] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1890] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1891] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1892] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1893] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states...

Claims

1. a means of receiving electronic files of reference materials; means for converting the contents of the received electronic file into text data using optical character recognition technology; A method for analyzing text data using a natural language processing engine, extracting keywords and themes, and generating metadata. means for storing the metadata; means for receiving a search prompt and extracting relevant keywords from the prompt; A means for searching the database based on the extracted keywords and picking up related materials; and means for extracting important sections from the picked-up material and providing them to the user.

2. means for generating and displaying thumbnails of important sections of the picked material; The system of claim 1 .

3. The means for analyzing the search prompt and extracting relevant keywords uses a machine learning model. The system of claim 1 .

Images