System

Abstract

A system is provided.SOLUTION: A system comprising: means for inputting a material; means for generating a feature vector from the material; means for storing the generated feature vector in a database; means for receiving a search request from a user; means for generating a feature vector of a query from the search request; means for comparing the feature vector of the query with feature vectors in the database; means for selecting a material with high relevance based on a comparison result; and means for displaying the selected material.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Companies and organizations create and store a large number of documents, but locating and using these documents takes time and effort. Furthermore, manually reviewing numerous documents to find relevant documents is inefficient. This reduces work efficiency and makes it difficult to quickly obtain the information you need. [Means for solving the problem]

[0005] This invention provides a system for efficiently managing materials and automatically suggesting optimal materials based on user requests. This system includes a means for inputting materials, a means for generating feature vectors from the materials, a means for storing the generated feature vectors in a database, a means for receiving a search request from a user, a means for generating a query feature vector from the search request, a means for comparing the feature vectors in the database with the query feature vector, a means for selecting highly relevant materials based on the comparison results, and a means for displaying the selected materials. The system also includes a function for storing metadata about materials and a function for analyzing queries using natural language processing technology, thereby enabling the system to suggest optimal materials in response to user requests.

[0006] "Materials" are digital content such as documents or presentations containing information.

[0007] A "feature vector" is a numerical representation of the content of a document or query, which makes it easier to analyze the characteristics of the document.

[0008] A "database" is a system or storage for efficiently managing and searching stored data and feature vectors.

[0009] A "search request" is a query or keyword related to the material a user wishes to find that is entered into the system.

[0010] A "query feature vector" is a numerical representation of a user's search request that the system uses to select appropriate materials.

[0011] "Natural language processing technology" is a technology for analyzing search requests entered by users, understanding their meaning, and processing them.

[0012] "Relevance" is an indicator of how relevant a document is to the search request, and serves as a criterion for the system to select appropriate documents.

[0013] "Metadata" is supplementary information about a material, including information such as the creation date and creator. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] As an embodiment of the present invention, a system is constructed as follows.

[0036] System configuration

[0037] The system consists of three main components: a server, a terminal, and a user. The server is responsible for managing materials and performing search functions, and the terminal provides an interface for user input and display of results. Users use the terminal to search for materials.

[0038] Program processing

[0039] Study and storage of materials

[0040] The server trains the generative AI model on new materials and stores them in storage.

[0041] 1. Inputting materials

[0042] The server receives text data of documents as input, such as business reports or marketing strategies.

[0043] 2. Generate feature vectors

[0044] The server uses a generative AI model to generate a feature vector from the document, which is a numerical representation of the document's content, making it easier to analyze the document's characteristics.

[0045] 3. Storage in the database

[0046] The server stores the generated feature vectors and the documents themselves in a database, along with metadata about the documents (creation date, creator, etc.).

[0047] Receiving requests from users

[0048] The user uses a dedicated terminal to input the image and keywords of the material to be searched.

[0049] 1. Enter your request

[0050] The user inputs a search query (keywords, sentences, etc.) into the input form of the search terminal. For example, the user inputs "materials related to marketing strategies for 2023."

[0051] 2. Sending the request

[0052] The terminal transmits the query entered by the user to the server.

[0053] Search for documents

[0054] The server analyzes the user's request and searches the database for the most suitable material.

[0055] 1. Parsing the query

[0056] The server analyzes the received query using natural language processing technology and understands the meaning of the query.

[0057] 2. Generate a feature vector for the query

[0058] The server generates a feature vector from the query, which is a numerical representation of the query's meaning.

[0059] 3. Comparison of materials

[0060] The server compares the feature vectors of the generated query with the feature vectors of the documents stored in the database.

[0061] 4. Selection of the most suitable materials

[0062] The server uses a similarity calculation (for example, cosine similarity) to select materials that are highly similar to the query.

[0063] Material suggestions

[0064] The server transmits a list of the selected materials to the user's terminal.

[0065] 1. Creating a list of materials

[0066] The server sorts the materials in descending order of relevance and creates a list.

[0067] 2. Submission of materials

[0068] The server transmits the material list to the user's terminal.

[0069] 3. View the list

[0070] The terminal visually displays the list of received materials to the user, including the title, summary, and creation date of each material.

[0071] 4. Check the details

[0072] The user selects a document in the list and checks its detailed contents.

[0073] Specific examples

[0074] For example, if a company provides a document titled "Marketing Strategy 2023" as input to the server, the server will train this document using a generative AI model and generate a feature vector. The generated feature vector and the document itself will then be stored in the database.

[0075] Next, the user enters "materials related to marketing strategies for 2023" as a search query into the terminal and submits it. The server analyzes this query and generates a feature vector. After comparing it with materials in the database, "Marketing strategies for 2023" in particular shows a high degree of relevance, so this material is selected.

[0076] Finally, a list of selected materials is sent to the user's device, where the user can review it. By selecting "Marketing Strategy 2023" from the list and checking the details, the user can efficiently obtain the information they are looking for.

[0077] This system allows users to quickly find relevant materials, improving work efficiency.

[0078] The processing flow will be explained below.

[0079] Program processing

[0080] Study and storage of materials

[0081] Step 1:

[0082] The server receives as input the text data of the materials, such as a PDF file or a presentation file about a marketing strategy.

[0083] Step 2:

[0084] The server passes the received documents to a generative AI model to generate a feature vector, which is a numerical representation of the document's content.

[0085] Step 3:

[0086] The server stores the generated feature vectors and the documents themselves in a database, along with the document metadata (creation date, creator, etc.).

[0087] Receiving requests from users

[0088] Step 4:

[0089] The user uses a dedicated terminal to input a query into an input form to search for materials, for example, "materials related to marketing strategies for 2023."

[0090] Step 5:

[0091] The terminal sends the entered query to the server, which sends the query as text data.

[0092] Search for documents

[0093] Step 6:

[0094] The server receives the query sent by the user and analyzes it using natural language processing techniques to accurately understand the content and meaning of the query.

[0095] Step 7:

[0096] The server generates a feature vector from the query, which is a numerical representation of the query content.

[0097] Step 8:

[0098] The server compares the feature vectors of the documents stored in the database with the feature vectors of the query, for example, by calculating the cosine similarity to evaluate the similarity.

[0099] Step 9:

[0100] The server selects the most similar documents and, based on the comparison results, chooses the document that best matches the query.

[0101] Material suggestions

[0102] Step 10:

[0103] The server lists the selected materials and sorts them in order of relevance. This list includes the title, summary, and creation date of each proposal.

[0104] Step 11:

[0105] The server sends the created list of materials to the user's terminal as text data.

[0106] Step 12:

[0107] The terminal visually displays a list of received materials to the user, allowing the user to check the title, summary, creation date, etc. at a glance.

[0108] Step 13:

[0109] The user selects a document in the list and performs an action to check the details, for example, clicking on a specific document to go to its details page.

[0110] Specific examples

[0111] For example, when a company provides a document called "Marketing Strategy 2023" as input to the server, the following steps are executed:

[0112] Step 1:

[0113] The server receives text data of a document called "Marketing Strategy 2023."

[0114] Step 2:

[0115] The server passes the material to a generative AI model, which generates a feature vector for the material.

[0116] Step 3:

[0117] The server stores the generated feature vectors and data in a database.

[0118] Step 4-5:

[0119] The user enters the query "materials related to marketing strategy for 2023" into the terminal, which then sends it to the server.

[0120] Steps 6-9:

[0121] The server analyzes the query, generates a feature vector of the query, compares it with materials in the database, and selects "Marketing Strategy 2023" as the most suitable material.

[0122] Steps 10-13:

[0123] The server generates a list of materials and transmits it to the terminal, which displays it, and the user selects a material from the list and checks the details.

[0124] The above processing steps enable users to efficiently search for and obtain the materials they need.

[0125] Example 1

[0126] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0127] With conventional document management systems, it was difficult to efficiently search and retrieve the desired information from a huge amount of documents. Furthermore, when users searched, documents that were not relevant to the query they entered were often displayed, reducing the user's work efficiency. Furthermore, searches often did not fully utilize the document metadata, making it difficult to perform highly accurate searches.

[0128] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0129] In this invention, the server includes means for inputting materials, means for generating a feature vector from the materials, means for storing the generated feature vector in a database, means for receiving a search request from a user, means for generating a query feature vector from the search request, means for comparing the feature vector in the database with the feature vector of the query, means for selecting highly relevant materials based on the comparison results, means for generating and transmitting a list of the selected materials, and means for displaying the selected materials. This makes it possible to quickly and accurately search for and obtain materials highly relevant to the search query entered by the user, thereby improving the user's work efficiency.

[0130] "Materials" refers to documents and data that record information, including, for example, business reports and marketing strategies.

[0131] A "feature vector" is a numerical representation of the content of a document or query, making its features easier to analyze.

[0132] A "database" is a management system for storing materials and feature vectors, and includes, for example, a relational database.

[0133] "User" means any individual or organization that uses the System to search and view materials.

[0134] "Search request" refers to a query or keywords entered by a user to search for materials.

[0135] A "query" is a keyword or sentence that a user inputs as a search request, and includes information to be searched.

[0136] "Natural language processing technology" is a technology that enables computers to understand and analyze human language, and includes processes such as text analysis, semantic analysis, and context understanding.

[0137] The "degree of relevance" is an index showing the relevance between the query and the material, and is a numerical value obtained by similarity calculation or the like.

[0138] A "list" is a list of selected materials, and is displayed visually to the user.

[0139] "Metadata" refers to additional information about a material, including information such as the creation date and creator.

[0140] MODE FOR CARRYING OUT THE INVENTION

[0141] This invention is directed to a system that is composed of three main elements: a server, a terminal, and a user. A specific embodiment for implementing the program for this system will be described below.

[0142] System configuration

[0143] The server is responsible for managing materials and search functions, providing functions such as inputting materials, generating feature vectors, storing them in a database, processing user search requests, analyzing queries, comparing materials, selecting the most suitable materials, creating a material list, and sending the material list. The server generates feature vectors for materials using a generative AI model (e.g., OpenAI's GPT-3 (registered trademark)) and analyzes queries using natural language processing technology. It also stores materials, feature vectors, and metadata using a database (e.g., MySQL (registered trademark) or PostgreSQL as a relational database).

[0144] The terminal provides an interface for user input and results display. The terminal receives the user's search request (query) and sends it to the server. The terminal also visually displays the list of materials received from the server to the user.

[0145] Users search for materials using a dedicated terminal. They input a search query through the terminal interface and check detailed materials from the list of materials presented as results.

[0146] Program processing explanation

[0147] The server receives the text data of the document and generates a feature vector from the document using a generative AI model (e.g., GPT-3). The generated feature vector is stored in a relational database, along with metadata about the document (such as creation date and author).

[0148] For example, if a company provides a document titled "Marketing Strategy 2023" to the server, the server will train the document using a generative AI model to generate a feature vector. This feature vector and the document itself will then be stored in the database.

[0149] A user inputs a search query using a dedicated terminal. For example, if a query such as "Marketing strategy materials for 2023" is input, the terminal sends the query to the server. The server then analyzes the received query using natural language processing technology and generates a feature vector to understand the meaning of the query.

[0150] The generated query feature vector is compared with the feature vectors of the documents in the database. This comparison uses a similarity calculation (e.g., cosine similarity). As a result, documents with high similarity to the query are selected.

[0151] The selected materials are then listed in descending order of relevance, and this list is sent to the user's device. The user's device visually displays the received materials list, allowing the user to select a material in the list and view its details.

[0152] Specific examples

[0153] For example, if a company provides a document titled "Marketing Strategy 2023" to the server, the server will train the document using a generative AI model to generate a feature vector. This feature vector and the document itself will then be stored in the database.

[0154] Next, the user enters "materials related to marketing strategies for 2023" as a search query into the terminal and submits it. The server analyzes this query and generates a feature vector. After comparing it with materials in the database, "Marketing strategies for 2023" in particular shows a high degree of relevance, so this material is selected.

[0155] Finally, a list of selected materials is sent to the user's device, where the user can review it. By selecting "Marketing Strategy 2023" from the list and checking the details, the user can efficiently obtain the information they are looking for.

[0156] Prompt Sentence Examples

[0157] "Find resources for 2023 marketing strategies."

[0158] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0159] System program processing flow

[0160] Step 1: Enter your materials

[0161] The server receives the text data of the document as input. For example, if a company provides a document called "Marketing Strategy 2023," the text data of this document is sent to the server. Specifically, the server receives the document in the form of an HTTP request or file upload and temporarily stores it in memory as input data.

[0162] Input: Text data of the document (e.g., "Marketing Strategy 2023")

[0163] Output: Data stored on the server

[0164] Step 2: Generate feature vectors

[0165] The server inputs the received material into a generative AI model (e.g., GPT-3) to generate a feature vector for the material. Specifically, the server passes the text of the material to the generative AI model, and the model extracts themes and keywords from the material to generate a feature vector.

[0166] Input: Text data of the material

[0167] Output: Feature vector of the material

[0168] Step 3: Store in the database

[0169] The server stores the generated feature vectors, the documents themselves, and metadata in a database. Specifically, the server uses SQL commands to save the document's feature vectors, text data, creation date and time, creator, and other metadata in a database (e.g., MySQL or PostgreSQL).

[0170] Input: feature vectors, text data, and metadata

[0171] Output: Material data and feature vectors stored in a database

[0172] Step 4: Entering Requests

[0173] A user inputs a search query using the interface of a dedicated terminal. For example, the user inputs the query "materials related to marketing strategies for 2023." Specifically, the user inputs the query into an input form using a web browser or a client application.

[0174] Input: Search query (e.g., "2023 marketing strategy materials")

[0175] Output: A search query is entered and persisted to the device.

[0176] Step 5: Submitting the request

[0177] The terminal sends the query entered by the user to the server. Specifically, the terminal sends an HTTP POST request and passes the query data to the server.

[0178] Input: search query

[0179] Output: Query data sent to the server

[0180] Step 6: Parsing the query

[0181] The server analyzes the received query using natural language processing technology to understand its meaning. Specifically, the server analyzes the query using a generative AI model to extract keywords and sentence structure.

[0182] Input: search query

[0183] Output: Query analysis results (extracted keywords, etc.)

[0184] Step 7: Generate a feature vector for the query

[0185] The server generates a feature vector from the query. Specifically, the server inputs the query text into a generative AI model and generates a feature vector that quantifies the meaning of the query.

[0186] Input: Query text data

[0187] Output: Query feature vector

[0188] Step 8: Compare materials

[0189] The server compares the feature vectors of the generated query with the feature vectors of the data stored in the database. Specifically, the server uses a similarity calculation algorithm (e.g., cosine similarity) to quickly compare the query and data vectors.

[0190] Input: feature vectors in the database, query feature vectors

[0191] Output: Similarity list between documents and queries

[0192] Step 9: Select the best materials

[0193] Based on the results of the similarity calculation, the server selects materials that have a high similarity to the query. Specifically, the server sorts the materials in descending order of similarity and selects the top-ranked materials.

[0194] Input: Similarity list between document and query

[0195] Output: List of highly relevant materials

[0196] Step 10: List your resources

[0197] The server sorts the documents in descending order of relevance and creates a list. Specifically, the server formats the selected documents in JSON format and prepares to return them as a response.

[0198] Input: Highly relevant materials

[0199] Output: Document list (JSON format)

[0200] Step 11: Submit your materials

[0201] The server sends the document list to the user's device. Specifically, the server generates an HTTP response and sends the document list to the device in JSON format.

[0202] Input: Document list (JSON format)

[0203] Output: List of materials sent to the user's terminal

[0204] Step 12: View the list

[0205] The device visually displays the list of received materials to the user, including the material's title, summary, creation date, etc. Specifically, the device parses the received JSON data and displays it on the user interface using HTML and CSS.

[0206] Input: List of received materials (JSON format)

[0207] Output: A visually displayed list of materials

[0208] Step 13: Check the details

[0209] The user selects a document from the list to view its details. When the user clicks on a document, the device opens a page displaying the details and then sends a request to the server to retrieve the full text of the document.

[0210] Input: User selection of material

[0211] Output: Detailed content of the document

[0212] (Application example 1)

[0213] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0214] Conventional search systems often have difficulty quickly and accurately finding the information and products users are looking for. In particular, virtual stores often contain a large amount of product information, making it difficult for users to easily find the products they are looking for. Furthermore, there is a lack of product suggestion systems that effectively utilize natural language processing technology, which means they are unable to accurately understand users' search intent and suggest appropriate products.

[0215] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0216] In this invention, the server includes means for inputting materials, means for generating a feature vector from the materials, means for storing the generated feature vector in a database, means for receiving a search request from a user, means for generating a query feature vector from the search request, means for comparing the feature vector in the database with the feature vector of the query, means for selecting highly relevant materials based on the comparison results, means for displaying the selected materials, means for receiving a product search request from a user using a smart device and analyzing the search request, means for quantifying the query feature vector using a generative AI model, and means for comparing the query feature vector with the feature vector in the product database by similarity calculation and suggesting highly relevant products. This enables the user to quickly and accurately search for and suggest products they are looking for in a virtual store, significantly improving the user's shopping experience.

[0217] "Materials" means information or data that is collected, stored or managed for a specific purpose.

[0218] A "feature vector" is a numerical representation of the content of a document or query, making it easier to analyze and compare.

[0219] A "database" is a structured collection of information and a system for storing, managing, and retrieving data.

[0220] "User" means any person or entity that utilizes the System to enter search requests and obtain results.

[0221] A "generative AI model" is a model that uses artificial intelligence technology to generate feature vectors from materials or queries.

[0222] A "smart device" is an electronic device that can connect to the Internet and has multiple functions (e.g., a smartphone or a head-mounted display).

[0223] "Natural language processing technology" is a technology that uses computers to understand and process human language (natural language).

[0224] A "search request" is an instruction or query for searching information that a user inputs to a system.

[0225] A "query" is a keyword or phrase that a user enters into a system to search for a specific piece of information.

[0226] "Similarity calculation" is a method for calculating the similarity between the feature vector of a query and the feature vectors in a database.

[0227] "Products" are objects for sale that are searched and suggested within the virtual store.

[0228] A "suggestion system" is a system that selects and displays appropriate materials and products based on a user's search request.

[0229] "Relevance" is an indicator of how closely a query matches materials or products in a database.

[0230] As an embodiment of the present invention, a system is constructed as follows. The system is composed of the main elements of a server, a terminal, and a user. The server is responsible for managing materials and products and for search functions, and the terminal provides an interface for user input and display of results. Users use the terminal to search for materials and products.

[0231] System configuration

[0232] server

[0233] The server has the following features:

[0234] 1. Input of materials and products

[0235] The server receives text data of materials and products (e.g., product names, descriptions, attributes, prices, etc.) as input.

[0236] 2. Generate feature vectors

[0237] The server generates feature vectors from documents and products using a generative AI model (e.g., a model using TENSORFLOW (registered trademark) or PyTorch).

[0238] 3. Storage in the database

[0239] The server stores the generated feature vectors and the materials and products themselves in a database, along with related metadata (such as registration date and time, category, etc.).

[0240] 4. Processing Search Requests

[0241] The server receives a search request from a user, analyzes the query using natural language processing techniques (e.g., spaCy or NLTK), and generates a corresponding feature vector.

[0242] 5. Similarity calculation

[0243] The server compares the feature vectors of the generated query with the feature vectors in the database using similarity calculations (for example, cosine similarity calculations using Scikit-learn) and selects materials and products with high matching scores.

[0244] 6. Submitting the results

[0245] The server transmits a list of the selected materials and products to the user's terminal and displays it visually.

[0246] Terminal

[0247] The terminal has the following features:

[0248] 1. Accepting input from the user

[0249] The terminal receives the user's search query (for example, "blue summer dress") from an input form and sends it to the server.

[0250] 2. Displaying the results

[0251] The terminal visually displays the search results received from the server to the user, including the product title, summary, price, and image.

[0252] User

[0253] A user uses the system in the following steps:

[0254] 1. Enter your search query

[0255] A user inputs a search query using a smart device (e.g., a smartphone or a head-mounted display).

[0256] 2. Check the search results

[0257] The user checks the search results displayed on the terminal, selects the required materials or products, and checks the details.

[0258] Specific examples

[0259] For example, when a user uses a smartphone to enter a search query such as "I want a blue summer dress," the device sends this query to a server. The server analyzes the query and generates a feature vector using a generative AI model. It compares this with the feature vectors of products stored in a database and displays the most suitable products (for example, "A lightweight, breathable blue summer dress") in a list ordered by relevance to the user. The user can then select a product from the results and view detailed information.

[0260] Prompt Sentence Examples

[0261] An example prompt for a generative AI model is:

[0262] Generate a feature vector based on the following product information: product name 'Blue summer dress', description 'Lightweight and breathable', attributes 'Color: Blue Season: Summer', price '3000 yen'.

[0263] In this way, users can quickly and accurately search for the materials and products they need and efficiently obtain the information they desire.

[0264] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0265] Step 1:

[0266] The server receives text data of materials and products as input. A user or administrator sends information about the materials and products (e.g., product name, description, attributes, price, etc.) to the server via an input device. This input data is in text format and includes a detailed description of the materials and products. The server prepares the received data to be passed to the generative AI model.

[0267] Step 2:

[0268] The server generates a feature vector from the received text data of the documents or products. Using a generative AI model (for example, a model using TensorFlow or PyTorch), the text data is converted into a feature vector. This feature vector is a numerical representation of the content of the documents or products, and is used for searching and comparison.

[0269] Step 3:

[0270] The server stores the generated feature vectors and the data of the materials and products themselves in a database. Related metadata (registration date and time, category, etc.) is also saved. This allows for quick retrieval of data in response to future search requests.

[0271] Step 4:

[0272] A user inputs a search query using a smart device (e.g., a smartphone or a head-mounted display). For example, the user inputs a query such as "I want a blue summer dress" into an input form. The device then sends this query to the server.

[0273] Step 5:

[0274] The server receives a search query from a user and analyzes it using natural language processing techniques (e.g., spaCy or NLTK). It extracts important keywords and phrases from the query and generates a corresponding feature vector, which helps understand the user's intent.

[0275] Step 6:

[0276] The server compares the feature vector of the generated query with the feature vectors in the database. It calculates the similarity between the query and the feature vectors of the materials or products using methods such as cosine similarity calculations using Scikit-learn. Based on this similarity, it selects materials or products with the highest matching scores.

[0277] Step 7:

[0278] The server generates a list of selected materials and products, sorts them by relevance, and sends the list to the user's device. The list includes product titles, summaries, prices, images, etc.

[0279] Step 8:

[0280] The terminal visually displays the list of materials and products received from the server to the user. The user can view the results and check detailed information as needed. Specifically, the user uses an interface that displays the product name, description, price, image, etc.

[0281] Step 9:

[0282] The user can select the desired product from the displayed list and check the detailed information. For example, they can view the detailed information of a "lightweight and breathable blue summer dress" and consider purchasing it.

[0283] Through the above processing steps, the user can quickly and accurately search for the desired materials or products and efficiently obtain the desired information.

[0284] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0285] As an embodiment of this invention, a system that combines document management and user emotion recognition is constructed. The system is composed of a server, a terminal, a user, and an emotion engine. The server is responsible for document management and search functions, while the terminal provides an interface for user input and result display. The user uses the terminal to search for documents, and the emotion engine is responsible for recognizing the user's emotions.

[0286] Program processing

[0287] Study and storage of materials

[0288] The server trains the generative AI model on new materials and stores them in storage.

[0289] 1. Inputting materials

[0290] The server receives text data of materials as input, such as PDF files or presentation files about marketing strategies.

[0291] 2. Generate feature vectors

[0292] The server uses a generative AI model to generate a feature vector from the document, which is a numerical representation of the document's content.

[0293] 3. Storage in the database

[0294] The server stores the generated feature vectors and the documents themselves in a database, along with the document metadata (creation date, creator, etc.).

[0295] Receiving requests from users

[0296] The user uses a dedicated terminal to input the image and keywords of the material to be searched.

[0297] 1. Enter your request

[0298] The user inputs a search query (keywords, sentences, etc.) into the input form of the search terminal. For example, the user inputs "materials related to marketing strategies for 2023."

[0299] 2. Sending the request

[0300] The terminal sends the entered query to the server, which sends the query as text data.

[0301] Emotion recognition

[0302] To recognize the user's emotions, the system is equipped with an emotion engine.

[0303] 1. Acquiring Emotion Data

[0304] When a user enters a query, the device captures the user's facial expressions and voice in real time through a camera, microphone, etc.

[0305] 2. Emotion Analysis

[0306] The server uses an emotion engine to analyze the captured data and recognize the user's emotions (e.g., joy, sadness, surprise, impatience, etc.).

[0307] Search for documents

[0308] The server analyzes the user's request and searches the database for the most suitable material.

[0309] 1. Parsing the query

[0310] The server analyzes the received query using natural language processing technology and understands the meaning of the query.

[0311] 2. Generate a feature vector for the query

[0312] The server generates a feature vector from the query, which is a numerical representation of the query content.

[0313] 3. Comparison of materials

[0314] The server compares the feature vectors of the documents stored in the database with the feature vectors of the query, for example, by calculating the cosine similarity to evaluate the similarity.

[0315] 4. Adjusting the fit

[0316] The server adjusts the selection criteria for materials based on the user's emotions recognized by the emotion engine. For example, if the user is in a hurry, the system will prioritize more intuitive and concise materials.

[0317] 5. Selection of the most suitable materials

[0318] The server selects materials with high similarity and creates an optimal list of materials that reflects the results of the emotion engine.

[0319] Material suggestions

[0320] The server transmits a list of the selected materials to the user's terminal.

[0321] 1. Creating a list of materials

[0322] The server sorts the documents in descending order of relevance and creates a list of them, including the title, summary, and creation date of each document.

[0323] 2. Submission of materials

[0324] The server sends the created list of materials to the user's terminal as text data.

[0325] 3. View the list

[0326] The terminal visually displays a list of received materials to the user, allowing the user to check the title, summary, creation date, etc. at a glance.

[0327] 4. Check the details

[0328] The user selects a document in the list and performs an action to check the details, for example, clicking on a specific document to go to its details page.

[0329] Specific examples

[0330] For example, when a company provides a document called "Marketing Strategy 2023" as input to the server, the following steps are executed:

[0331] Steps 1-3:

[0332] The server trains a generative AI model on this material, generates a feature vector for the material, and stores it in a database.

[0333] Step 4-5:

[0334] The user enters the query "materials related to marketing strategy for 2023" into the terminal, which then sends it to the server.

[0335] Steps 6-7:

[0336] The server analyzes the query, generates a feature vector for the query, and compares it with the material in the database.

[0337] Step 8:

[0338] The device captures the user's facial expressions and voice through a camera and microphone, and the server analyzes the user's emotions using an emotion engine. For example, if the user is feeling anxious, the system will adjust the materials it selects based on this information.

[0339] Steps 9-13:

[0340] The server selects highly relevant materials, including "Marketing Strategy 2023," generates a list of materials, and sends it to the user's device. The device displays the list, and the user can select a material from the list to check its details.

[0341] This system allows users to efficiently search and obtain the materials they need, and also allows them to receive optimal suggestions based on their emotions.

[0342] The processing flow will be explained below.

[0343] Program processing

[0344] Study and storage of materials

[0345] Step 1:

[0346] The server receives as input the text data of the document, for example a PDF file or a presentation file about a new marketing strategy.

[0347] Step 2:

[0348] The server passes the received documents to the generative AI model, which generates a feature vector from the documents. This feature vector is a numerical representation of the contents of the documents.

[0349] Step 3:

[0350] The server stores the generated feature vectors in a database, as well as the materials themselves and their metadata (creation date, creator, etc.).

[0351] Receiving requests from users

[0352] Step 4:

[0353] Users use a dedicated terminal to input images and keywords for the materials they are searching for, for example, "materials related to marketing strategies for 2023."

[0354] Step 5:

[0355] The terminal sends the entered query to the server, which sends the query as text data.

[0356] Emotion recognition

[0357] Step 6:

[0358] The device captures the user's facial expressions and voice data in real time through a camera and microphone while the user enters a query.

[0359] Step 7:

[0360] The server uses an emotion engine to analyze the captured data and recognize the user's emotions, for example, whether the user is anxious or calm.

[0361] Search for documents

[0362] Step 8:

[0363] The server analyzes the received query using natural language processing technology to understand its meaning, clarifying the meaning of keywords and phrases.

[0364] Step 9:

[0365] The server generates a feature vector from the query, which is a numerical representation of the query content.

[0366] Step 10:

[0367] The server compares the feature vectors in the database with the feature vectors of the query, for example by calculating cosine similarity to assess relevance.

[0368] Step 11:

[0369] The server adjusts the selection criteria for materials based on the user's emotions recognized by the emotion engine. For example, if the user is in a hurry, the server adjusts the selection criteria to prioritize intuitive and concise materials.

[0370] Step 12:

[0371] The server selects highly relevant materials and lists the most suitable materials based on the similarity between the query and the feature vectors of the materials.

[0372] Material suggestions

[0373] Step 13:

[0374] The server creates a list of the selected materials, sorted by relevance, including title, summary, and creation date.

[0375] Step 14:

[0376] The server sends the created list of materials to the user's terminal as text data.

[0377] Step 15:

[0378] The terminal visually displays the list of received materials to the user, including the title, summary, and creation date.

[0379] Step 16:

[0380] The user can select an item in the list to view its details, for example, by clicking on a specific item to go to its details page.

[0381] Specific examples

[0382] For example, when a company provides a document called "Marketing Strategy 2023" as input to the server, the following steps are executed:

[0383] Step 1:

[0384] The server receives text data of a document called "Marketing Strategy 2023."

[0385] Step 2:

[0386] The server passes the material to a generative AI model to generate a feature vector.

[0387] Step 3:

[0388] The server stores the generated feature vectors, the material itself, and its metadata in a database.

[0389] Step 4:

[0390] The user enters the query "Marketing strategy materials for 2023" into the terminal.

[0391] Step 5:

[0392] The terminal sends this query to the server.

[0393] Step 6:

[0394] The device uses a camera and microphone to capture the user's emotional data.

[0395] Step 7:

[0396] The server analyzes the user's emotions using an emotion engine and recognizes that the user is, for example, "anxious."

[0397] Steps 8-12:

[0398] The server analyzes the query, generates a feature vector, compares it with the database, and adjusts the selection criteria based on the user's sentiment to select the most suitable list of materials.

[0399] Steps 13-16:

[0400] The server generates a list of materials and sends it to the user's terminal, which displays it. The user selects a material and checks its details.

[0401] This system allows users to efficiently search and obtain the materials they need, and also allows them to receive optimal suggestions based on their emotions.

[0402] Example 2

[0403] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0404] Existing document management systems search and suggest documents without considering the user's emotions, making it difficult to intuitively provide the documents the user needs. In particular, the optimal documents may differ depending on the user's emotional state, which can lead to a poor user experience.

[0405] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for inputting materials, means for generating a feature vector from the materials, means for storing the generated feature vector in a database, means for receiving a search request from a user, means for generating a query feature vector from the search request, means for comparing the feature vector in the database with the feature vector of the query, means for selecting highly suitable materials based on the comparison results, means for displaying the selected materials, means for recognizing the user's emotions, and means for adjusting the material selection criteria based on the emotions. This makes it possible to search for and suggest optimal materials according to the user's emotional state.

[0406] "Materials" refers to documents or data in which information is organized in the form of text, images, audio, etc.

[0407] A "feature vector" is vector data that numerically represents the content of a document or the meaning of a query.

[0408] A "database" is a system that systematically stores and manages materials, feature vectors, metadata, etc.

[0409] A "user search request" refers to keywords or sentences that a user inputs to the system to search for specific materials.

[0410] A "query feature vector" is numerical vector data generated based on a search request.

[0411] The "comparison result" refers to the evaluation result of similarity or compatibility obtained by comparing the feature vectors in the database with the feature vectors of the query.

[0412] "Material selection criteria" refers to the criteria or indicators used to select materials, including sentiment and relevance.

[0413] "User's emotions" refers to the user's psychological or emotional state (for example, joy, sadness, surprise, impatience, etc.).

[0414] "Camera and microphone" refers to input devices for capturing the user's facial expressions and voice.

[0415] A "generative AI model" is a generative model that uses artificial intelligence and is an algorithm used to generate feature vectors from materials or queries.

[0416] MODE FOR CARRYING OUT THE INVENTION

[0417] As an embodiment of this invention, we will build a system that combines document management and user emotion recognition. The system is composed of a server, a terminal, a user, and an emotion engine. The server is responsible for document management and search functions, while the terminal provides an interface for user input and display of results. The user uses the terminal to search for documents, and the emotion engine is responsible for recognizing the user's emotions. Specific embodiments of each element of this system will be described below.

[0418] Study and storage of materials

[0419] The server trains new materials using a generative AI model. Specifically, the server receives the text data of the materials as input and generates feature vectors using a generative AI model (e.g., GPT-4 (registered trademark)). This feature vector is a numerical representation of the content of the materials and is stored in a database. The stored data includes not only the feature vectors, but also the materials themselves and their metadata (creation date and time, creator, etc.).

[0420] Receiving requests from users

[0421] A user attempts to search for materials using a dedicated terminal. The user enters a search query (e.g., "materials related to marketing strategies for 2023") into the terminal's input form, and the terminal sends this query as text data to the server. The query is sent to the server as an HTTP request and processed by the server.

[0422] Emotion recognition

[0423] When a user enters a query, the device captures the user's facial expressions and voice in real time through a camera and microphone. This data is sent to a server, which then uses an emotion engine to analyze the user's emotions. For example, the device uses emotion analysis tools from Microsoft's Azure Face API or Google Cloud Natural Language to recognize the user's emotions (such as joy, sadness, surprise, and impatience).

[0424] Search for documents

[0425] The server analyzes the user's request and searches the database for the most appropriate materials. First, it analyzes the received query using natural language processing technology (e.g., BERT or GPT-4) to understand the meaning of the query. Then, it uses a generative AI model to generate a feature vector from the query. The server compares this feature vector with feature vectors in the database and evaluates the similarity, for example, by calculating cosine similarity. Furthermore, it adjusts the material selection criteria based on the user's emotions recognized by the emotion engine, and selects materials with high similarity.

[0426] Material suggestions

[0427] The server sends a list of selected materials to the user's device. The list is sorted in order of relevance and includes information such as the material's title, summary, and creation date. The device visually displays the received material list, and the user can select a material from the list to check its details. Specifically, clicking on the list on the device takes the user to the details page.

[0428] Specific examples

[0429] For example, if a company provides a document titled "Marketing Strategy 2023" as input, the server trains the document through a generative AI model, generates a feature vector, and stores it in a database. The user then enters a query, "Documents related to marketing strategy for 2023," and the device sends this to the server. The server analyzes the query, generates a specific feature vector, and compares it with documents in the database. The device captures the user's facial expressions and voice and analyzes their emotions using an emotion engine. For example, if the user is anxious, the system adjusts the documents it selects based on this information. Finally, the server selects documents with high relevance, including "Marketing Strategy 2023," generates a document list, and sends it to the device. The user can select a document from the list to view its details.

[0430] This system allows users to efficiently search and obtain the materials they need, and also allows them to receive optimal suggestions based on their emotions.

[0431] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0432] Step 1:

[0433] Inputting materials

[0434] The server receives text data of documents as input, such as PDF files or presentation files, which may be obtained from an internal management system or an external data source.

[0435] Input: Text data of the material (e.g., PDF file, Word file)

[0436] What it does: The server reads the document and extracts its contents as a string.

[0437] Output: Text data

[0438] Step 2:

[0439] Feature Vector Generation

[0440] The server uses a generative AI model (e.g., GPT-4) to generate a feature vector from the document. The feature vector is a numerical representation of the content of the document.

[0441] Input: Extracted text data

[0442] How it works: Text data is fed into a generative AI model to generate a feature vector.

[0443] Output: feature vector

[0444] Step 3:

[0445] Storage in the database

[0446] The server stores the generated feature vectors and the materials themselves in a database together with metadata.

[0447] Input: feature vector, text data of the material, metadata (creation date, creator, etc.)

[0448] Operation: Issues an INSERT statement to the database and saves the data.

[0449] Output: Data is stored in the database

[0450] Step 4:

[0451] Entering a Request

[0452] The user enters a search query (for example, "materials related to marketing strategies for 2023") into the input form on the dedicated terminal.

[0453] Input: User's search query (keywords, sentences)

[0454] Action: Enter text using the keyboard on the device and click the send button.

[0455] Output: Text data of the search query

[0456] Step 5:

[0457] Submitting a request

[0458] The device sends the entered query to the server, sending data using an HTTP request.

[0459] Input: Text data of the search query

[0460] What it does: Sends a query to the server via an HTTP POST request.

[0461] Output: Query data sent to the server

[0462] Step 6:

[0463] Acquiring emotion data

[0464] When a user enters a query, the device captures the user's facial expressions and voice in real time through the camera and microphone.

[0465] Input: User's facial expression data, voice data

[0466] What it does: Captures a picture of your face with a webcam and records audio with a microphone.

[0467] Output: Captured facial expression data, audio data

[0468] Step 7:

[0469] Emotion Analysis

[0470] The server uses an emotion engine (e.g., Azure Face API) to analyze the captured data and recognize the user's emotions.

[0471] Input: Captured facial expression data, voice data

[0472] Operation: Facial images and voice data are input into the emotion engine to obtain emotional information.

[0473] Output: Emotion information (happiness, sadness, surprise, impatience, etc.)

[0474] Step 8:

[0475] Query Analysis

[0476] The server analyzes the received query using natural language processing technology (e.g., BERT or GPT-4) to understand the meaning of the query.

[0477] Input: Text data of the search query

[0478] How it works: Inputs query text into a natural language processing engine to perform analysis.

[0479] Output: Semantic information of the parsed query

[0480] Step 9:

[0481] Generate a query feature vector

[0482] The server generates a feature vector from the query.

[0483] Input: Parsed query semantics

[0484] How it works: Query information is input into the generative AI model, which generates a feature vector.

[0485] Output: Feature vector of the query

[0486] Step 10:

[0487] Comparison of materials

[0488] The server compares the feature vectors of the documents stored in the database with the feature vectors of the query, for example, by calculating the cosine similarity to evaluate the similarity.

[0489] Input: Query feature vector, database feature vector

[0490] Operation: Performs a cosine similarity calculation on each feature vector.

[0491] Output: Similarity score with each document

[0492] Step 11:

[0493] Adjusting the fit

[0494] The server adjusts the selection criteria for materials based on the user's emotions recognized by the emotion engine. If the user is in a hurry, intuitive and concise materials will be suggested first.

[0495] Input: Similarity score, emotion information

[0496] How it works: Adjust filtering conditions and score weighting based on emotional information.

[0497] Output: Adjusted similarity score

[0498] Step 12:

[0499] Selection of the most suitable materials

[0500] The server selects materials with high similarity and creates an optimal list of materials that reflects the results of the emotion engine.

[0501] Input: Adjusted similarity score

[0502] Operation: Select high-scoring materials and create a list.

[0503] Output: List of best materials

[0504] Step 13:

[0505] Creating a list of materials

[0506] The server sorts the documents in descending order of relevance and creates a list, including the document title, summary, creation date, etc.

[0507] Input: List of optimal materials

[0508] What it does: Sorts the list of documents and adds titles, summaries, creation dates, etc.

[0509] Output: Document list

[0510] Step 14:

[0511] Sending materials

[0512] The server sends the created document list to the user's terminal, and sends text data as an HTTP response.

[0513] Input: Document list

[0514] Operation: Returns the list of materials as an HTTP response in JSON format or similar.

[0515] Output: List of materials sent to the terminal

[0516] Step 15:

[0517] Viewing the list

[0518] The terminal visually displays a list of received materials to the user, allowing the user to check the title, summary, creation date, etc. at a glance.

[0519] Input: Document list

[0520] What it does: Renders a list visually with HTML and CSS.

[0521] Output: List of displayed materials

[0522] Step 16:

[0523] Check details

[0524] The user selects a document in the list and performs an action to check the details, for example, clicking on a specific document to go to its details page.

[0525] Input: Selection action on the material list

[0526] Action: Click on the document link to display the detailed view.

[0527] Output: Show detailed information

[0528] (Application example 2)

[0529] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0530] Conventional document management systems have the problem that when a user searches, the optimal document is not presented based on the user's emotions, making it difficult to effectively retrieve documents depending on the user's emotional state. Furthermore, search results may not fully satisfy the user's needs, resulting in a poor user experience. This is particularly important in virtual store environments, where users are required to quickly and accurately obtain products and information that match their emotions.

[0531] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting materials, means for generating a feature vector from the materials, means for storing the generated feature vector in a database, means for receiving a search request from a user, means for generating a query feature vector from the search request, means for comparing the feature vector in the database with the feature vector of the query, means for selecting materials with high suitability based on the comparison results, means for displaying the selected materials, means for recognizing the user's emotions, and means for adjusting the suitability based on the user's emotions. This makes it possible to quickly search for and suggest optimal materials and products that match the user's emotions, thereby improving the user experience.

[0532] "Materials" refers to information or data that users search or view.

[0533] A "feature vector" is vector-format data that quantifies the content of a document or query and characterizes that content.

[0534] A "database" is a collection of stored data that stores feature vectors and information for later retrieval and comparison.

[0535] A "search request" is a query or keyword entered by a user to search for a particular resource.

[0536] A "query" is a character string or a sentence in a natural language that a user uses to make a search request.

[0537] "Emotion" refers to a psychological state that can be recognized from a user's facial expression or voice, and includes states such as joy, sadness, surprise, and impatience.

[0538] "Relevance" is an index that indicates the relevance or match of a document to a user's search request or emotional state.

[0539] A "server" is a central computer that manages materials, performs searches, performs sentiment analysis, etc., and operates the entire system in conjunction with terminals.

[0540] A "terminal" is a device that provides an interface for users to search for and view materials.

[0541] An "emotion engine" is software or hardware that analyzes a user's emotions and recognizes emotions from facial expressions, voice, gestures, etc.

[0542] As an embodiment of the present invention, a system that combines document management and user emotion recognition is constructed. This system is composed of the following elements: a server, a terminal, a user, and an emotion engine.

[0543] 1. Server

[0544] The server is responsible for document management and search functions. It performs a series of processes, including inputting new documents, generating feature vectors, storing them in the database, and analyzing search requests. Specifically, it receives the document's text data, generates feature vectors using a generative AI model, and stores these in the database. It also receives search queries from users, analyzes them using natural language processing technology, and generates a query feature vector. It then compares the query feature vector with the feature vectors in the database to select documents with a high degree of relevance. During this process, the relevance is adjusted based on the user's emotions recognized by the emotion engine. By adjusting the relevance, it selects documents that best fit the user's emotional state.

[0545] 2. Terminal

[0546] The terminal provides an interface for users to input materials and make search requests. Users use the terminal to search for materials and display search results. The terminal captures the user's facial expressions and voice in real time through a camera and microphone and sends them to the server. When a user searches for materials, they enter a search query into the terminal's input form, which is sent to the server. The terminal visually displays the search results sent from the server, and the user can select the appropriate material from the list.

[0547] 3. Emotion Engine

[0548] The emotion engine recognizes the user's emotions. Specifically, when a user uses a device to input a query, it analyzes data captured through the device's camera and microphone to recognize the user's emotions. This allows the system to understand the user's emotional state. For example, if the user is in a hurry, the system will prioritize suggesting concise materials that can be understood immediately.

[0549] Specific examples

[0550] For example, in a VR shopping application for a virtual store, a user might say, "I'm looking for new sneakers." In this case, the device captures the user's facial expressions and voice in real time and sends them to the server. The server uses an emotion engine to analyze the user's emotions, and if it determines that the user is feeling anxious, it adjusts the suitability based on that information. As a result, sneakers that are most suitable for the user (popular items, highly reviewed items, etc.) are displayed preferentially.

[0551] Example prompts for generative AI models

[0552] "We suggest the best products for users who are in a hurry to find new sneakers."

[0553] This system allows users to efficiently search for and obtain the products they need in a VR environment and receive optimal suggestions based on their emotions at the time. The required hardware includes a VR headset, camera, and microphone. A typical VR headset (e.g., Meta Quest 2 or HTC Vive) is used as the specific device. Emotion analysis uses libraries such as Hume AI and Affectiva as emotion engines. On the server side, natural language processing technology (e.g., SpaCy and BERT) is used for document management and search.

[0554] In this way, the system of the invention can quickly search for and suggest the most suitable materials and products according to the user's emotional state, improving the user experience.

[0555] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0556] Step 1:

[0557] The user wears a VR headset and accesses a virtual store. The device captures the user's voice input, gaze, and facial expressions. As input, it acquires the user's voice query (e.g., "I'm looking for new sneakers") and emotional data including facial expressions and vocal tone.

[0558] Step 2:

[0559] The device extracts a search query from the voice input and sends it to the emotion engine. Specifically, it uses a microphone to convert the voice into text, which is then used as the query. The query is then sent to the server as text data.

[0560] Step 3:

[0561] The server analyzes the input query using natural language processing techniques (e.g., SpaCy, BERT) and generates a feature vector for the query. It receives the text data of the query as input and outputs the feature vector.

[0562] Step 4:

[0563] The device captures emotion data from the user's facial expressions and voice in real time and sends it to the emotion engine. Specifically, it uses a camera and microphone to capture the user's facial expressions and voice and sends the data to the server. The emotion data is output as extracted features.

[0564] Step 5:

[0565] The server analyzes the emotion data acquired by the emotion engine and recognizes the user's emotional state. It receives emotion data as input and outputs an emotional state such as joy, sadness, surprise, or impatience.

[0566] Step 6:

[0567] The server compares the query feature vector with the feature vectors in the database and calculates the similarity. Specifically, it uses mathematical techniques such as cosine similarity to compare the feature vectors of each document, and outputs the most suitable document from the database.

[0568] Step 7:

[0569] The server adjusts the relevance of search results based on the user's emotional state. For example, if the user is in a hurry, it will prioritize concise and intuitive materials. It receives the user's emotional state and the similarity of the materials as input, and outputs a list of materials with adjusted relevance.

[0570] Step 8:

[0571] The server selects the most relevant materials, creates a list of materials, and sends it to the user's terminal. The server receives the adjusted list of materials as input and sends it to the terminal as text data.

[0572] Step 9:

[0573] The device visually displays the received list of materials to the user. Specifically, the list is displayed on a display in the VR environment, and the user can select the appropriate material from the list. The output shows the title and summary of the material.

[0574] Step 10:

[0575] The user selects the document for which they wish to view details from the displayed list of documents. They are then taken to the details page of the selected document and can view the contents of the document. The system receives the user's selection as input and displays the detailed contents of the document as output.

[0576] Through this series of steps, users can efficiently search for and acquire the products they need in a VR environment and receive optimal suggestions based on their emotions at the time.

[0577] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0578] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0579] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0580] [Second embodiment]

[0581] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0582] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0583] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0584] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0585] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0586] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0587] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0588] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0589] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0590] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0591] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0592] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0593] As an embodiment of the present invention, a system is constructed as follows.

[0594] System configuration

[0595] The system consists of three main components: a server, a terminal, and a user. The server is responsible for managing materials and performing search functions, and the terminal provides an interface for user input and display of results. Users use the terminal to search for materials.

[0596] Program processing

[0597] Study and storage of materials

[0598] The server trains the generative AI model on new materials and stores them in storage.

[0599] 1. Inputting materials

[0600] The server receives text data of documents as input, such as business reports or marketing strategies.

[0601] 2. Generate feature vectors

[0602] The server uses a generative AI model to generate a feature vector from the document, which is a numerical representation of the document's content, making it easier to analyze the document's characteristics.

[0603] 3. Storage in the database

[0604] The server stores the generated feature vectors and the documents themselves in a database, along with metadata about the documents (creation date, creator, etc.).

[0605] Receiving requests from users

[0606] The user uses a dedicated terminal to input the image and keywords of the material to be searched.

[0607] 1. Enter your request

[0608] The user inputs a search query (keywords, sentences, etc.) into the input form of the search terminal. For example, the user inputs "materials related to marketing strategies for 2023."

[0609] 2. Sending the request

[0610] The terminal transmits the query entered by the user to the server.

[0611] Search for documents

[0612] The server analyzes the user's request and searches the database for the most suitable material.

[0613] 1. Parsing the query

[0614] The server analyzes the received query using natural language processing technology and understands the meaning of the query.

[0615] 2. Generate a feature vector for the query

[0616] The server generates a feature vector from the query, which is a numerical representation of the query's meaning.

[0617] 3. Comparison of materials

[0618] The server compares the feature vectors of the generated query with the feature vectors of the documents stored in the database.

[0619] 4. Selection of the most suitable materials

[0620] The server uses a similarity calculation (for example, cosine similarity) to select materials that are highly similar to the query.

[0621] Material suggestions

[0622] The server transmits a list of the selected materials to the user's terminal.

[0623] 1. Creating a list of materials

[0624] The server sorts the materials in descending order of relevance and creates a list.

[0625] 2. Submission of materials

[0626] The server transmits the material list to the user's terminal.

[0627] 3. View the list

[0628] The terminal visually displays the list of received materials to the user, including the title, summary, and creation date of each material.

[0629] 4. Check the details

[0630] The user selects a document in the list and checks its detailed contents.

[0631] Specific examples

[0632] For example, if a company provides a document titled "Marketing Strategy 2023" as input to the server, the server will train this document using a generative AI model and generate a feature vector. The generated feature vector and the document itself will then be stored in the database.

[0633] Next, the user enters "materials related to marketing strategies for 2023" as a search query into the terminal and submits it. The server analyzes this query and generates a feature vector. After comparing it with materials in the database, "Marketing strategies for 2023" in particular shows a high degree of relevance, so this material is selected.

[0634] Finally, a list of selected materials is sent to the user's device, where the user can review it. By selecting "Marketing Strategy 2023" from the list and checking the details, the user can efficiently obtain the information they are looking for.

[0635] This system allows users to quickly find relevant materials, improving work efficiency.

[0636] The processing flow will be explained below.

[0637] Program processing

[0638] Study and storage of materials

[0639] Step 1:

[0640] The server receives as input the text data of the materials, such as a PDF file or a presentation file about a marketing strategy.

[0641] Step 2:

[0642] The server passes the received documents to a generative AI model to generate a feature vector, which is a numerical representation of the document's content.

[0643] Step 3:

[0644] The server stores the generated feature vectors and the documents themselves in a database, along with the document metadata (creation date, creator, etc.).

[0645] Receiving requests from users

[0646] Step 4:

[0647] The user uses a dedicated terminal to input a query into an input form to search for materials, for example, "materials related to marketing strategies for 2023."

[0648] Step 5:

[0649] The terminal sends the entered query to the server, which sends the query as text data.

[0650] Search for documents

[0651] Step 6:

[0652] The server receives the query sent by the user and analyzes it using natural language processing techniques to accurately understand the content and meaning of the query.

[0653] Step 7:

[0654] The server generates a feature vector from the query, which is a numerical representation of the query content.

[0655] Step 8:

[0656] The server compares the feature vectors of the documents stored in the database with the feature vectors of the query, for example, by calculating the cosine similarity to evaluate the similarity.

[0657] Step 9:

[0658] The server selects the most similar documents and, based on the comparison results, chooses the document that best matches the query.

[0659] Material suggestions

[0660] Step 10:

[0661] The server lists the selected materials and sorts them in order of relevance. This list includes the title, summary, and creation date of each proposal.

[0662] Step 11:

[0663] The server sends the created list of materials to the user's terminal as text data.

[0664] Step 12:

[0665] The terminal visually displays a list of received materials to the user, allowing the user to check the title, summary, creation date, etc. at a glance.

[0666] Step 13:

[0667] The user selects a document in the list and performs an action to check the details, for example, clicking on a specific document to go to its details page.

[0668] Specific examples

[0669] For example, when a company provides a document called "Marketing Strategy 2023" as input to the server, the following steps are executed:

[0670] Step 1:

[0671] The server receives text data of a document called "Marketing Strategy 2023."

[0672] Step 2:

[0673] The server passes the material to a generative AI model, which generates a feature vector for the material.

[0674] Step 3:

[0675] The server stores the generated feature vectors and data in a database.

[0676] Step 4-5:

[0677] The user enters the query "materials related to marketing strategy for 2023" into the terminal, which then sends it to the server.

[0678] Steps 6-9:

[0679] The server analyzes the query, generates a feature vector of the query, compares it with materials in the database, and selects "Marketing Strategy 2023" as the most suitable material.

[0680] Steps 10-13:

[0681] The server generates a list of materials and transmits it to the terminal, which displays it, and the user selects a material from the list and checks the details.

[0682] The above processing steps enable users to efficiently search for and obtain the materials they need.

[0683] Example 1

[0684] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0685] With conventional document management systems, it was difficult to efficiently search and retrieve the desired information from a huge amount of documents. Furthermore, when users searched, documents that were not relevant to the query they entered were often displayed, reducing the user's work efficiency. Furthermore, searches often did not fully utilize the document metadata, making it difficult to perform highly accurate searches.

[0686] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0687] In this invention, the server includes means for inputting materials, means for generating a feature vector from the materials, means for storing the generated feature vector in a database, means for receiving a search request from a user, means for generating a query feature vector from the search request, means for comparing the feature vector in the database with the feature vector of the query, means for selecting highly relevant materials based on the comparison results, means for generating and transmitting a list of the selected materials, and means for displaying the selected materials. This makes it possible to quickly and accurately search for and obtain materials highly relevant to the search query entered by the user, thereby improving the user's work efficiency.

[0688] "Materials" refers to documents and data that record information, including, for example, business reports and marketing strategies.

[0689] A "feature vector" is a numerical representation of the content of a document or query, making its features easier to analyze.

[0690] A "database" is a management system for storing materials and feature vectors, and includes, for example, a relational database.

[0691] "User" means any individual or organization that uses the System to search and view materials.

[0692] "Search request" refers to a query or keywords entered by a user to search for materials.

[0693] A "query" is a keyword or sentence that a user inputs as a search request, and includes information to be searched.

[0694] "Natural language processing technology" is a technology that enables computers to understand and analyze human language, and includes processes such as text analysis, semantic analysis, and context understanding.

[0695] The "degree of relevance" is an index showing the relevance between the query and the material, and is a numerical value obtained by similarity calculation or the like.

[0696] A "list" is a list of selected materials, and is displayed visually to the user.

[0697] "Metadata" refers to additional information about a material, including information such as the creation date and creator.

[0698] MODE FOR CARRYING OUT THE INVENTION

[0699] This invention is directed to a system that is composed of three main elements: a server, a terminal, and a user. A specific embodiment for implementing the program for this system will be described below.

[0700] System configuration

[0701] The server is responsible for managing materials and search functions, providing functions such as inputting materials, generating feature vectors, storing them in a database, processing user search requests, analyzing queries, comparing materials, selecting the most suitable materials, creating a material list, and sending the material list. The server generates material feature vectors using a generative AI model (e.g., OpenAI's GPT-3) and analyzes queries using natural language processing technology. It also uses a database (e.g., MySQL or PostgreSQL, which are relational databases) to store materials, feature vectors, and metadata.

[0702] The terminal provides an interface for user input and results display. The terminal receives the user's search request (query) and sends it to the server. The terminal also visually displays the list of materials received from the server to the user.

[0703] Users search for materials using a dedicated terminal. They input a search query through the terminal interface and check detailed materials from the list of materials presented as results.

[0704] Program processing explanation

[0705] The server receives the text data of the document and generates a feature vector from the document using a generative AI model (e.g., GPT-3). The generated feature vector is stored in a relational database, along with metadata about the document (such as creation date and author).

[0706] For example, if a company provides a document titled "Marketing Strategy 2023" to the server, the server will train the document using a generative AI model to generate a feature vector. This feature vector and the document itself will then be stored in the database.

[0707] A user inputs a search query using a dedicated terminal. For example, if a query such as "Marketing strategy materials for 2023" is input, the terminal sends the query to the server. The server then analyzes the received query using natural language processing technology and generates a feature vector to understand the meaning of the query.

[0708] The generated query feature vector is compared with the feature vectors of the documents in the database. This comparison uses a similarity calculation (e.g., cosine similarity). As a result, documents with high similarity to the query are selected.

[0709] The selected materials are then listed in descending order of relevance, and this list is sent to the user's device. The user's device visually displays the received materials list, allowing the user to select a material in the list and view its details.

[0710] Specific examples

[0711] For example, if a company provides a document titled "Marketing Strategy 2023" to the server, the server will train the document using a generative AI model to generate a feature vector. This feature vector and the document itself will then be stored in the database.

[0712] Next, the user enters "materials related to marketing strategies for 2023" as a search query into the terminal and submits it. The server analyzes this query and generates a feature vector. After comparing it with materials in the database, "Marketing strategies for 2023" in particular shows a high degree of relevance, so this material is selected.

[0713] Finally, a list of selected materials is sent to the user's device, where the user can review it. By selecting "Marketing Strategy 2023" from the list and checking the details, the user can efficiently obtain the information they are looking for.

[0714] Prompt Sentence Examples

[0715] "Find resources for 2023 marketing strategies."

[0716] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0717] System program processing flow

[0718] Step 1: Enter your materials

[0719] The server receives the text data of the document as input. For example, if a company provides a document called "Marketing Strategy 2023," the text data of this document is sent to the server. Specifically, the server receives the document in the form of an HTTP request or file upload and temporarily stores it in memory as input data.

[0720] Input: Text data of the document (e.g., "Marketing Strategy 2023")

[0721] Output: Data stored on the server

[0722] Step 2: Generate feature vectors

[0723] The server inputs the received material into a generative AI model (e.g., GPT-3) to generate a feature vector for the material. Specifically, the server passes the text of the material to the generative AI model, and the model extracts themes and keywords from the material to generate a feature vector.

[0724] Input: Text data of the material

[0725] Output: Feature vector of the material

[0726] Step 3: Store in the database

[0727] The server stores the generated feature vectors, the documents themselves, and metadata in a database. Specifically, the server uses SQL commands to save the document's feature vectors, text data, creation date and time, creator, and other metadata in a database (e.g., MySQL or PostgreSQL).

[0728] Input: feature vectors, text data, and metadata

[0729] Output: Material data and feature vectors stored in a database

[0730] Step 4: Entering Requests

[0731] A user inputs a search query using the interface of a dedicated terminal. For example, the user inputs the query "materials related to marketing strategies for 2023." Specifically, the user inputs the query into an input form using a web browser or a client application.

[0732] Input: Search query (e.g., "2023 marketing strategy materials")

[0733] Output: A search query is entered and persisted to the device.

[0734] Step 5: Submitting the request

[0735] The terminal sends the query entered by the user to the server. Specifically, the terminal sends an HTTP POST request and passes the query data to the server.

[0736] Input: search query

[0737] Output: Query data sent to the server

[0738] Step 6: Parsing the query

[0739] The server analyzes the received query using natural language processing technology to understand its meaning. Specifically, the server analyzes the query using a generative AI model to extract keywords and sentence structure.

[0740] Input: search query

[0741] Output: Query analysis results (extracted keywords, etc.)

[0742] Step 7: Generate a feature vector for the query

[0743] The server generates a feature vector from the query. Specifically, the server inputs the query text into a generative AI model and generates a feature vector that quantifies the meaning of the query.

[0744] Input: Query text data

[0745] Output: Query feature vector

[0746] Step 8: Compare materials

[0747] The server compares the feature vectors of the generated query with the feature vectors of the data stored in the database. Specifically, the server uses a similarity calculation algorithm (e.g., cosine similarity) to quickly compare the query and data vectors.

[0748] Input: feature vectors in the database, query feature vectors

[0749] Output: Similarity list between documents and queries

[0750] Step 9: Select the best materials

[0751] Based on the results of the similarity calculation, the server selects materials that have a high similarity to the query. Specifically, the server sorts the materials in descending order of similarity and selects the top-ranked materials.

[0752] Input: Similarity list between document and query

[0753] Output: List of highly relevant materials

[0754] Step 10: List your resources

[0755] The server sorts the documents in descending order of relevance and creates a list. Specifically, the server formats the selected documents in JSON format and prepares to return them as a response.

[0756] Input: Highly relevant materials

[0757] Output: Document list (JSON format)

[0758] Step 11: Submit your materials

[0759] The server sends the document list to the user's device. Specifically, the server generates an HTTP response and sends the document list to the device in JSON format.

[0760] Input: Document list (JSON format)

[0761] Output: List of materials sent to the user's terminal

[0762] Step 12: View the list

[0763] The device visually displays the list of received materials to the user, including the material's title, summary, creation date, etc. Specifically, the device parses the received JSON data and displays it on the user interface using HTML and CSS.

[0764] Input: List of received materials (JSON format)

[0765] Output: A visually displayed list of materials

[0766] Step 13: Check the details

[0767] The user selects a document from the list to view its details. When the user clicks on a document, the device opens a page displaying the details and then sends a request to the server to retrieve the full text of the document.

[0768] Input: User selection of material

[0769] Output: Detailed content of the document

[0770] (Application example 1)

[0771] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0772] Conventional search systems often have difficulty quickly and accurately finding the information and products users are looking for. In particular, virtual stores often contain a large amount of product information, making it difficult for users to easily find the products they are looking for. Furthermore, there is a lack of product suggestion systems that effectively utilize natural language processing technology, which means they are unable to accurately understand users' search intent and suggest appropriate products.

[0773] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0774] In this invention, the server includes means for inputting materials, means for generating a feature vector from the materials, means for storing the generated feature vector in a database, means for receiving a search request from a user, means for generating a query feature vector from the search request, means for comparing the feature vector in the database with the feature vector of the query, means for selecting highly relevant materials based on the comparison results, means for displaying the selected materials, means for receiving a product search request from a user using a smart device and analyzing the search request, means for quantifying the query feature vector using a generative AI model, and means for comparing the query feature vector with the feature vector in the product database by similarity calculation and suggesting highly relevant products. This enables the user to quickly and accurately search for and suggest products they are looking for in a virtual store, significantly improving the user's shopping experience.

[0775] "Materials" means information or data that is collected, stored or managed for a specific purpose.

[0776] A "feature vector" is a numerical representation of the content of a document or query, making it easier to analyze and compare.

[0777] A "database" is a structured collection of information and a system for storing, managing, and retrieving data.

[0778] "User" means any person or entity that utilizes the System to enter search requests and obtain results.

[0779] A "generative AI model" is a model that uses artificial intelligence technology to generate feature vectors from materials or queries.

[0780] A "smart device" is an electronic device that can connect to the Internet and has multiple functions (e.g., a smartphone or a head-mounted display).

[0781] "Natural language processing technology" is a technology that uses computers to understand and process human language (natural language).

[0782] A "search request" is an instruction or query for searching information that a user inputs to a system.

[0783] A "query" is a keyword or phrase that a user enters into a system to search for a specific piece of information.

[0784] "Similarity calculation" is a method for calculating the similarity between the feature vector of a query and the feature vectors in a database.

[0785] "Products" are objects for sale that are searched and suggested within the virtual store.

[0786] A "suggestion system" is a system that selects and displays appropriate materials and products based on a user's search request.

[0787] "Relevance" is an indicator of how closely a query matches materials or products in a database.

[0788] As an embodiment of the present invention, a system is constructed as follows. The system is composed of the main elements of a server, a terminal, and a user. The server is responsible for managing materials and products and for search functions, and the terminal provides an interface for user input and display of results. Users use the terminal to search for materials and products.

[0789] System configuration

[0790] server

[0791] The server has the following features:

[0792] 1. Input of materials and products

[0793] The server receives text data of materials and products (e.g., product names, descriptions, attributes, prices, etc.) as input.

[0794] 2. Generate feature vectors

[0795] The server generates feature vectors from documents and products using a generative AI model (e.g., a model using TensorFlow or PyTorch).

[0796] 3. Storage in the database

[0797] The server stores the generated feature vectors and the materials and products themselves in a database, along with related metadata (such as registration date and time, category, etc.).

[0798] 4. Processing Search Requests

[0799] The server receives a search request from a user, analyzes the query using natural language processing techniques (e.g., spaCy or NLTK), and generates a corresponding feature vector.

[0800] 5. Similarity calculation

[0801] The server compares the feature vectors of the generated query with the feature vectors in the database using similarity calculations (for example, cosine similarity calculations using Scikit-learn) and selects materials and products with high matching scores.

[0802] 6. Submitting the results

[0803] The server transmits a list of the selected materials and products to the user's terminal and displays it visually.

[0804] Terminal

[0805] The terminal has the following features:

[0806] 1. Accepting input from the user

[0807] The terminal receives the user's search query (for example, "blue summer dress") from an input form and sends it to the server.

[0808] 2. Displaying the results

[0809] The terminal visually displays the search results received from the server to the user, including the product title, summary, price, and image.

[0810] User

[0811] A user uses the system in the following steps:

[0812] 1. Enter your search query

[0813] A user inputs a search query using a smart device (e.g., a smartphone or a head-mounted display).

[0814] 2. Check the search results

[0815] The user checks the search results displayed on the terminal, selects the required materials or products, and checks the details.

[0816] Specific examples

[0817] For example, when a user uses a smartphone to enter a search query such as "I want a blue summer dress," the device sends this query to a server. The server analyzes the query and generates a feature vector using a generative AI model. It compares this with the feature vectors of products stored in a database and displays the most suitable products (for example, "A lightweight, breathable blue summer dress") in a list ordered by relevance to the user. The user can then select a product from the results and view detailed information.

[0818] Prompt Sentence Examples

[0819] An example prompt for a generative AI model is:

[0820] Generate a feature vector based on the following product information: product name 'Blue summer dress', description 'Lightweight and breathable', attributes 'Color: Blue Season: Summer', price '3000 yen'.

[0821] In this way, users can quickly and accurately search for the materials and products they need and efficiently obtain the information they desire.

[0822] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0823] Step 1:

[0824] The server receives text data of materials and products as input. A user or administrator sends information about the materials and products (e.g., product name, description, attributes, price, etc.) to the server via an input device. This input data is in text format and includes a detailed description of the materials and products. The server prepares the received data to be passed to the generative AI model.

[0825] Step 2:

[0826] The server generates a feature vector from the received text data of the documents or products. Using a generative AI model (for example, a model using TensorFlow or PyTorch), the text data is converted into a feature vector. This feature vector is a numerical representation of the content of the documents or products, and is used for searching and comparison.

[0827] Step 3:

[0828] The server stores the generated feature vectors and the data of the materials and products themselves in a database. Related metadata (registration date and time, category, etc.) is also saved. This allows for quick retrieval of data in response to future search requests.

[0829] Step 4:

[0830] A user inputs a search query using a smart device (e.g., a smartphone or a head-mounted display). For example, the user inputs a query such as "I want a blue summer dress" into an input form. The device then sends this query to the server.

[0831] Step 5:

[0832] The server receives a search query from a user and analyzes it using natural language processing techniques (e.g., spaCy or NLTK). It extracts important keywords and phrases from the query and generates a corresponding feature vector, which helps understand the user's intent.

[0833] Step 6:

[0834] The server compares the feature vector of the generated query with the feature vectors in the database. It calculates the similarity between the query and the feature vectors of the materials or products using methods such as cosine similarity calculations using Scikit-learn. Based on this similarity, it selects materials or products with the highest matching scores.

[0835] Step 7:

[0836] The server generates a list of selected materials and products, sorts them by relevance, and sends the list to the user's device. The list includes product titles, summaries, prices, images, etc.

[0837] Step 8:

[0838] The terminal visually displays the list of materials and products received from the server to the user. The user can view the results and check detailed information as needed. Specifically, the user uses an interface that displays the product name, description, price, image, etc.

[0839] Step 9:

[0840] The user can select the desired product from the displayed list and check the detailed information. For example, they can view the detailed information of a "lightweight and breathable blue summer dress" and consider purchasing it.

[0841] Through the above processing steps, the user can quickly and accurately search for the desired materials or products and efficiently obtain the desired information.

[0842] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0843] As an embodiment of this invention, a system that combines document management and user emotion recognition is constructed. The system is composed of a server, a terminal, a user, and an emotion engine. The server is responsible for document management and search functions, while the terminal provides an interface for user input and result display. The user uses the terminal to search for documents, and the emotion engine is responsible for recognizing the user's emotions.

[0844] Program processing

[0845] Study and storage of materials

[0846] The server trains the generative AI model on new materials and stores them in storage.

[0847] 1. Inputting materials

[0848] The server receives text data of materials as input, such as PDF files or presentation files about marketing strategies.

[0849] 2. Generate feature vectors

[0850] The server uses a generative AI model to generate a feature vector from the document, which is a numerical representation of the document's content.

[0851] 3. Storage in the database

[0852] The server stores the generated feature vectors and the documents themselves in a database, along with the document metadata (creation date, creator, etc.).

[0853] Receiving requests from users

[0854] The user uses a dedicated terminal to input the image and keywords of the material to be searched.

[0855] 1. Enter your request

[0856] The user inputs a search query (keywords, sentences, etc.) into the input form of the search terminal. For example, the user inputs "materials related to marketing strategies for 2023."

[0857] 2. Sending the request

[0858] The terminal sends the entered query to the server, which sends the query as text data.

[0859] Emotion recognition

[0860] To recognize the user's emotions, the system is equipped with an emotion engine.

[0861] 1. Acquiring Emotion Data

[0862] When a user enters a query, the device captures the user's facial expressions and voice in real time through a camera, microphone, etc.

[0863] 2. Emotion Analysis

[0864] The server uses an emotion engine to analyze the captured data and recognize the user's emotions (e.g., joy, sadness, surprise, impatience, etc.).

[0865] Search for documents

[0866] The server analyzes the user's request and searches the database for the most suitable material.

[0867] 1. Parsing the query

[0868] The server analyzes the received query using natural language processing technology and understands the meaning of the query.

[0869] 2. Generate a feature vector for the query

[0870] The server generates a feature vector from the query, which is a numerical representation of the query content.

[0871] 3. Comparison of materials

[0872] The server compares the feature vectors of the documents stored in the database with the feature vectors of the query, for example, by calculating the cosine similarity to evaluate the similarity.

[0873] 4. Adjusting the fit

[0874] The server adjusts the selection criteria for materials based on the user's emotions recognized by the emotion engine. For example, if the user is in a hurry, the system will prioritize more intuitive and concise materials.

[0875] 5. Selection of the most suitable materials

[0876] The server selects materials with high similarity and creates an optimal list of materials that reflects the results of the emotion engine.

[0877] Material suggestions

[0878] The server transmits a list of the selected materials to the user's terminal.

[0879] 1. Creating a list of materials

[0880] The server sorts the documents in descending order of relevance and creates a list of them, including the title, summary, and creation date of each document.

[0881] 2. Submission of materials

[0882] The server sends the created list of materials to the user's terminal as text data.

[0883] 3. View the list

[0884] The terminal visually displays a list of received materials to the user, allowing the user to check the title, summary, creation date, etc. at a glance.

[0885] 4. Check the details

[0886] The user selects a document in the list and performs an action to check the details, for example, clicking on a specific document to go to its details page.

[0887] Specific examples

[0888] For example, when a company provides a document called "Marketing Strategy 2023" as input to the server, the following steps are executed:

[0889] Steps 1-3:

[0890] The server trains a generative AI model on this material, generates a feature vector for the material, and stores it in a database.

[0891] Step 4-5:

[0892] The user enters the query "materials related to marketing strategy for 2023" into the terminal, which then sends it to the server.

[0893] Steps 6-7:

[0894] The server analyzes the query, generates a feature vector for the query, and compares it with the material in the database.

[0895] Step 8:

[0896] The device captures the user's facial expressions and voice through a camera and microphone, and the server analyzes the user's emotions using an emotion engine. For example, if the user is feeling anxious, the system will adjust the materials it selects based on this information.

[0897] Steps 9-13:

[0898] The server selects highly relevant materials, including "Marketing Strategy 2023," generates a list of materials, and sends it to the user's device. The device displays the list, and the user can select a material from the list to check its details.

[0899] This system allows users to efficiently search and obtain the materials they need, and also allows them to receive optimal suggestions based on their emotions.

[0900] The processing flow will be explained below.

[0901] Program processing

[0902] Study and storage of materials

[0903] Step 1:

[0904] The server receives as input the text data of the document, for example a PDF file or a presentation file about a new marketing strategy.

[0905] Step 2:

[0906] The server passes the received documents to the generative AI model, which generates a feature vector from the documents. This feature vector is a numerical representation of the contents of the documents.

[0907] Step 3:

[0908] The server stores the generated feature vectors in a database, as well as the materials themselves and their metadata (creation date, creator, etc.).

[0909] Receiving requests from users

[0910] Step 4:

[0911] Users use a dedicated terminal to input images and keywords for the materials they are searching for, for example, "materials related to marketing strategies for 2023."

[0912] Step 5:

[0913] The terminal sends the entered query to the server, which sends the query as text data.

[0914] Emotion recognition

[0915] Step 6:

[0916] The device captures the user's facial expressions and voice data in real time through a camera and microphone while the user enters a query.

[0917] Step 7:

[0918] The server uses an emotion engine to analyze the captured data and recognize the user's emotions, for example, whether the user is anxious or calm.

[0919] Search for documents

[0920] Step 8:

[0921] The server analyzes the received query using natural language processing technology to understand its meaning, clarifying the meaning of keywords and phrases.

[0922] Step 9:

[0923] The server generates a feature vector from the query, which is a numerical representation of the query content.

[0924] Step 10:

[0925] The server compares the feature vectors in the database with the feature vectors of the query, for example by calculating cosine similarity to assess relevance.

[0926] Step 11:

[0927] The server adjusts the selection criteria for materials based on the user's emotions recognized by the emotion engine. For example, if the user is in a hurry, the server adjusts the selection criteria to prioritize intuitive and concise materials.

[0928] Step 12:

[0929] The server selects highly relevant materials and lists the most suitable materials based on the similarity between the query and the feature vectors of the materials.

[0930] Material suggestions

[0931] Step 13:

[0932] The server creates a list of the selected materials, sorted by relevance, including title, summary, and creation date.

[0933] Step 14:

[0934] The server sends the created list of materials to the user's terminal as text data.

[0935] Step 15:

[0936] The terminal visually displays the list of received materials to the user, including the title, summary, and creation date.

[0937] Step 16:

[0938] The user can select an item in the list to view its details, for example, by clicking on a specific item to go to its details page.

[0939] Specific examples

[0940] For example, when a company provides a document called "Marketing Strategy 2023" as input to the server, the following steps are executed:

[0941] Step 1:

[0942] The server receives text data of a document called "Marketing Strategy 2023."

[0943] Step 2:

[0944] The server passes the material to a generative AI model to generate a feature vector.

[0945] Step 3:

[0946] The server stores the generated feature vectors, the material itself, and its metadata in a database.

[0947] Step 4:

[0948] The user enters the query "Marketing strategy materials for 2023" into the terminal.

[0949] Step 5:

[0950] The terminal sends this query to the server.

[0951] Step 6:

[0952] The device uses a camera and microphone to capture the user's emotional data.

[0953] Step 7:

[0954] The server analyzes the user's emotions using an emotion engine and recognizes that the user is, for example, "anxious."

[0955] Steps 8-12:

[0956] The server analyzes the query, generates a feature vector, compares it with the database, and adjusts the selection criteria based on the user's sentiment to select the most suitable list of materials.

[0957] Steps 13-16:

[0958] The server generates a list of materials and sends it to the user's terminal, which displays it. The user selects a material and checks its details.

[0959] This system allows users to efficiently search and obtain the materials they need, and also allows them to receive optimal suggestions based on their emotions.

[0960] Example 2

[0961] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0962] Existing document management systems search and suggest documents without considering the user's emotions, making it difficult to intuitively provide the documents the user needs. In particular, the optimal documents may differ depending on the user's emotional state, which can lead to a poor user experience.

[0963] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for inputting materials, means for generating a feature vector from the materials, means for storing the generated feature vector in a database, means for receiving a search request from a user, means for generating a query feature vector from the search request, means for comparing the feature vector in the database with the feature vector of the query, means for selecting highly suitable materials based on the comparison results, means for displaying the selected materials, means for recognizing the user's emotions, and means for adjusting the material selection criteria based on the emotions. This makes it possible to search for and suggest optimal materials according to the user's emotional state.

[0964] "Materials" refers to documents or data in which information is organized in the form of text, images, audio, etc.

[0965] A "feature vector" is vector data that numerically represents the content of a document or the meaning of a query.

[0966] A "database" is a system that systematically stores and manages materials, feature vectors, metadata, etc.

[0967] A "user search request" refers to keywords or sentences that a user inputs to the system to search for specific materials.

[0968] A "query feature vector" is numerical vector data generated based on a search request.

[0969] The "comparison result" refers to the evaluation result of similarity or compatibility obtained by comparing the feature vectors in the database with the feature vectors of the query.

[0970] "Material selection criteria" refers to the criteria or indicators used to select materials, including sentiment and relevance.

[0971] "User's emotions" refers to the user's psychological or emotional state (for example, joy, sadness, surprise, impatience, etc.).

[0972] "Camera and microphone" refers to input devices for capturing the user's facial expressions and voice.

[0973] A "generative AI model" is a generative model that uses artificial intelligence and is an algorithm used to generate feature vectors from materials or queries.

[0974] MODE FOR CARRYING OUT THE INVENTION

[0975] As an embodiment of this invention, we will build a system that combines document management and user emotion recognition. The system is composed of a server, a terminal, a user, and an emotion engine. The server is responsible for document management and search functions, while the terminal provides an interface for user input and display of results. The user uses the terminal to search for documents, and the emotion engine is responsible for recognizing the user's emotions. Specific embodiments of each element of this system will be described below.

[0976] Study and storage of materials

[0977] The server trains new documents using a generative AI model. Specifically, the server receives the document's text data as input and generates a feature vector using a generative AI model (e.g., GPT-4). This feature vector is a numerical representation of the document's content and is stored in a database. The stored data includes not only the feature vector, but also the document itself and its metadata (creation date and time, creator, etc.).

[0978] Receiving requests from users

[0979] A user attempts to search for materials using a dedicated terminal. The user enters a search query (e.g., "materials related to marketing strategies for 2023") into the terminal's input form, and the terminal sends this query as text data to the server. The query is sent to the server as an HTTP request and processed by the server.

[0980] Emotion recognition

[0981] When a user enters a query, the device captures the user's facial expressions and voice in real time through a camera and microphone. This data is sent to a server, which then uses an emotion engine to analyze the user's emotions. For example, it uses Microsoft's Azure Face API or Google Cloud Natural Language emotion analysis tools to recognize the user's emotions (happiness, sadness, surprise, impatience, etc.).

[0982] Search for documents

[0983] The server analyzes the user's request and searches the database for the most appropriate materials. First, it analyzes the received query using natural language processing technology (e.g., BERT or GPT-4) to understand the meaning of the query. Then, it uses a generative AI model to generate a feature vector from the query. The server compares this feature vector with feature vectors in the database and evaluates the similarity, for example, by calculating cosine similarity. Furthermore, it adjusts the material selection criteria based on the user's emotions recognized by the emotion engine, and selects materials with high similarity.

[0984] Material suggestions

[0985] The server sends a list of selected materials to the user's device. The list is sorted in order of relevance and includes information such as the material's title, summary, and creation date. The device visually displays the received material list, and the user can select a material from the list to check its details. Specifically, clicking on the list on the device takes the user to the details page.

[0986] Specific examples

[0987] For example, if a company provides a document titled "Marketing Strategy 2023" as input, the server trains the document through a generative AI model, generates a feature vector, and stores it in a database. The user then enters a query, "Documents related to marketing strategy for 2023," and the device sends this to the server. The server analyzes the query, generates a specific feature vector, and compares it with documents in the database. The device captures the user's facial expressions and voice and analyzes their emotions using an emotion engine. For example, if the user is anxious, the system adjusts the documents it selects based on this information. Finally, the server selects documents with high relevance, including "Marketing Strategy 2023," generates a document list, and sends it to the device. The user can select a document from the list to view its details.

[0988] This system allows users to efficiently search and obtain the materials they need, and also allows them to receive optimal suggestions based on their emotions.

[0989] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0990] Step 1:

[0991] Inputting materials

[0992] The server receives text data of documents as input, such as PDF files or presentation files, which may be obtained from an internal management system or an external data source.

[0993] Input: Text data of the material (e.g., PDF file, Word file)

[0994] What it does: The server reads the document and extracts its contents as a string.

[0995] Output: Text data

[0996] Step 2:

[0997] Feature Vector Generation

[0998] The server uses a generative AI model (e.g., GPT-4) to generate a feature vector from the document. The feature vector is a numerical representation of the content of the document.

[0999] Input: Extracted text data

[1000] How it works: Text data is fed into a generative AI model to generate a feature vector.

[1001] Output: Feature vector

[1002] Step 3:

[1003] Storage in the database

[1004] The server stores the generated feature vectors and the materials themselves in a database together with metadata.

[1005] Input: feature vector, text data of the material, metadata (creation date, creator, etc.)

[1006] Operation: Issues an INSERT statement to the database and saves the data.

[1007] Output: Data is stored in the database

[1008] Step 4:

[1009] Entering a Request

[1010] The user enters a search query (for example, "materials related to marketing strategies for 2023") into the input form on the dedicated terminal.

[1011] Input: User's search query (keywords, sentences)

[1012] Action: Enter text using the keyboard on the device and click the send button.

[1013] Output: Text data of the search query

[1014] Step 5:

[1015] Submitting a request

[1016] The device sends the entered query to the server, sending data using an HTTP request.

[1017] Input: Text data of the search query

[1018] What it does: Sends a query to the server via an HTTP POST request.

[1019] Output: Query data sent to the server

[1020] Step 6:

[1021] Acquiring emotion data

[1022] When a user enters a query, the device captures the user's facial expressions and voice in real time through the camera and microphone.

[1023] Input: User's facial expression data, voice data

[1024] What it does: Captures a picture of your face with a webcam and records audio with a microphone.

[1025] Output: Captured facial expression data, audio data

[1026] Step 7:

[1027] Emotion Analysis

[1028] The server uses an emotion engine (e.g., Azure Face API) to analyze the captured data and recognize the user's emotions.

[1029] Input: Captured facial expression data, voice data

[1030] Operation: Facial images and voice data are input into the emotion engine to obtain emotional information.

[1031] Output: Emotion information (happiness, sadness, surprise, impatience, etc.)

[1032] Step 8:

[1033] Query Analysis

[1034] The server analyzes the received query using natural language processing technology (e.g., BERT or GPT-4) to understand the meaning of the query.

[1035] Input: Text data of the search query

[1036] How it works: Inputs query text into a natural language processing engine to perform analysis.

[1037] Output: Semantic information of the parsed query

[1038] Step 9:

[1039] Generate a query feature vector

[1040] The server generates a feature vector from the query.

[1041] Input: Parsed query semantics

[1042] How it works: Query information is input into the generative AI model, which generates a feature vector.

[1043] Output: Feature vector of the query

[1044] Step 10:

[1045] Comparison of materials

[1046] The server compares the feature vectors of the documents stored in the database with the feature vectors of the query, for example, by calculating the cosine similarity to evaluate the similarity.

[1047] Input: Query feature vector, database feature vector

[1048] Operation: Performs a cosine similarity calculation on each feature vector.

[1049] Output: Similarity score with each document

[1050] Step 11:

[1051] Adjusting the fit

[1052] The server adjusts the selection criteria for materials based on the user's emotions recognized by the emotion engine. If the user is in a hurry, intuitive and concise materials will be suggested first.

[1053] Input: Similarity score, emotion information

[1054] How it works: Adjust filtering conditions and score weighting based on emotional information.

[1055] Output: Adjusted similarity score

[1056] Step 12:

[1057] Selection of the most suitable materials

[1058] The server selects materials with high similarity and creates an optimal list of materials that reflects the results of the emotion engine.

[1059] Input: Adjusted similarity score

[1060] Operation: Select high-scoring materials and create a list.

[1061] Output: List of best materials

[1062] Step 13:

[1063] Creating a list of materials

[1064] The server sorts the documents in descending order of relevance and creates a list, including the document title, summary, creation date, etc.

[1065] Input: List of optimal materials

[1066] What it does: Sorts the list of documents and adds titles, summaries, creation dates, etc.

[1067] Output: Document list

[1068] Step 14:

[1069] Sending materials

[1070] The server sends the created document list to the user's terminal, and sends text data as an HTTP response.

[1071] Input: Document list

[1072] Operation: Returns the list of materials as an HTTP response in JSON format or similar.

[1073] Output: List of materials sent to the terminal

[1074] Step 15:

[1075] Viewing the list

[1076] The terminal visually displays a list of received materials to the user, allowing the user to check the title, summary, creation date, etc. at a glance.

[1077] Input: Document list

[1078] What it does: Renders a list visually with HTML and CSS.

[1079] Output: List of displayed materials

[1080] Step 16:

[1081] Check details

[1082] The user selects a document in the list and performs an action to check the details, for example, clicking on a specific document to go to its details page.

[1083] Input: Selection action on the material list

[1084] Action: Click on the document link to display the detailed view.

[1085] Output: Show detailed information

[1086] (Application example 2)

[1087] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1088] Conventional document management systems have the problem that when a user searches, the optimal document is not presented based on the user's emotions, making it difficult to effectively retrieve documents depending on the user's emotional state. Furthermore, search results may not fully satisfy the user's needs, resulting in a poor user experience. This is particularly important in virtual store environments, where users are required to quickly and accurately obtain products and information that match their emotions.

[1089] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting materials, means for generating a feature vector from the materials, means for storing the generated feature vector in a database, means for receiving a search request from a user, means for generating a query feature vector from the search request, means for comparing the feature vector in the database with the feature vector of the query, means for selecting materials with high suitability based on the comparison results, means for displaying the selected materials, means for recognizing the user's emotions, and means for adjusting the suitability based on the user's emotions. This makes it possible to quickly search for and suggest optimal materials and products that match the user's emotions, thereby improving the user experience.

[1090] "Materials" refers to information or data that users search or view.

[1091] A "feature vector" is vector-format data that quantifies the content of a document or query and characterizes that content.

[1092] A "database" is a collection of stored data that stores feature vectors and information for later retrieval and comparison.

[1093] A "search request" is a query or keyword entered by a user to search for a particular resource.

[1094] A "query" is a character string or a sentence in a natural language that a user uses to make a search request.

[1095] "Emotion" refers to a psychological state that can be recognized from a user's facial expression or voice, and includes states such as joy, sadness, surprise, and impatience.

[1096] "Relevance" is an index that indicates the relevance or match of a document to a user's search request or emotional state.

[1097] A "server" is a central computer that manages materials, performs searches, performs sentiment analysis, etc., and operates the entire system in conjunction with terminals.

[1098] A "terminal" is a device that provides an interface for users to search for and view materials.

[1099] An "emotion engine" is software or hardware that analyzes a user's emotions and recognizes emotions from facial expressions, voice, gestures, etc.

[1100] As an embodiment of the present invention, a system that combines document management and user emotion recognition is constructed. This system is composed of the following elements: a server, a terminal, a user, and an emotion engine.

[1101] 1. Server

[1102] The server is responsible for document management and search functions. It performs a series of processes, including inputting new documents, generating feature vectors, storing them in the database, and analyzing search requests. Specifically, it receives the document's text data, generates feature vectors using a generative AI model, and stores these in the database. It also receives search queries from users, analyzes them using natural language processing technology, and generates a query feature vector. It then compares the query feature vector with the feature vectors in the database to select documents with a high degree of relevance. During this process, the relevance is adjusted based on the user's emotions recognized by the emotion engine. By adjusting the relevance, it selects documents that best fit the user's emotional state.

[1103] 2. Terminal

[1104] The terminal provides an interface for users to input materials and make search requests. Users use the terminal to search for materials and display search results. The terminal captures the user's facial expressions and voice in real time through a camera and microphone and sends them to the server. When a user searches for materials, they enter a search query into the terminal's input form, which is sent to the server. The terminal visually displays the search results sent from the server, and the user can select the appropriate material from the list.

[1105] 3. Emotion Engine

[1106] The emotion engine recognizes the user's emotions. Specifically, when a user uses a device to input a query, it analyzes data captured through the device's camera and microphone to recognize the user's emotions. This allows the system to understand the user's emotional state. For example, if the user is in a hurry, the system will prioritize suggesting concise materials that can be understood immediately.

[1107] Specific examples

[1108] For example, in a VR shopping application for a virtual store, a user might say, "I'm looking for new sneakers." In this case, the device captures the user's facial expressions and voice in real time and sends them to the server. The server uses an emotion engine to analyze the user's emotions, and if it determines that the user is feeling anxious, it adjusts the suitability based on that information. As a result, sneakers that are most suitable for the user (popular items, highly reviewed items, etc.) are displayed preferentially.

[1109] Example prompts for generative AI models

[1110] "We suggest the best products for users who are in a hurry to find new sneakers."

[1111] This system allows users to efficiently search for and obtain the products they need in a VR environment and receive optimal suggestions based on their emotions at the time. The required hardware includes a VR headset, camera, and microphone. A typical VR headset (e.g., Meta Quest 2 or HTC Vive) is used as the specific device. Emotion analysis uses libraries such as Hume AI and Affectiva as emotion engines. On the server side, natural language processing technology (e.g., SpaCy and BERT) is used for document management and search.

[1112] In this way, the system of the invention can quickly search for and suggest the most suitable materials and products according to the user's emotional state, improving the user experience.

[1113] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1114] Step 1:

[1115] The user wears a VR headset and accesses a virtual store. The device captures the user's voice input, gaze, and facial expressions. As input, it acquires the user's voice query (e.g., "I'm looking for new sneakers") and emotional data including facial expressions and vocal tone.

[1116] Step 2:

[1117] The device extracts a search query from the voice input and sends it to the emotion engine. Specifically, it uses a microphone to convert the voice into text, which is then used as the query. The query is then sent to the server as text data.

[1118] Step 3:

[1119] The server analyzes the input query using natural language processing techniques (e.g., SpaCy, BERT) and generates a feature vector for the query. It receives the text data of the query as input and outputs the feature vector.

[1120] Step 4:

[1121] The device captures emotion data from the user's facial expressions and voice in real time and sends it to the emotion engine. Specifically, it uses a camera and microphone to capture the user's facial expressions and voice and sends the data to the server. The emotion data is output as extracted features.

[1122] Step 5:

[1123] The server analyzes the emotion data acquired by the emotion engine and recognizes the user's emotional state. It receives emotion data as input and outputs an emotional state such as joy, sadness, surprise, or impatience.

[1124] Step 6:

[1125] The server compares the query feature vector with the feature vectors in the database and calculates the similarity. Specifically, it uses mathematical techniques such as cosine similarity to compare the feature vectors of each document, and outputs the most suitable document from the database.

[1126] Step 7:

[1127] The server adjusts the relevance of search results based on the user's emotional state. For example, if the user is in a hurry, it will prioritize concise and intuitive materials. It receives the user's emotional state and the similarity of the materials as input, and outputs a list of materials with adjusted relevance.

[1128] Step 8:

[1129] The server selects the most relevant materials, creates a list of materials, and sends it to the user's terminal. The server receives the adjusted list of materials as input and sends it to the terminal as text data.

[1130] Step 9:

[1131] The device visually displays the received list of materials to the user. Specifically, the list is displayed on a display in the VR environment, and the user can select the appropriate material from the list. The output shows the title and summary of the material.

[1132] Step 10:

[1133] The user selects the document for which they wish to view details from the displayed list of documents. They are then taken to the details page of the selected document and can view the contents of the document. The system receives the user's selection as input and displays the detailed contents of the document as output.

[1134] Through this series of steps, users can efficiently search for and acquire the products they need in a VR environment and receive optimal suggestions based on their emotions at the time.

[1135] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1136] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1137] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1138] [Third embodiment]

[1139] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1140] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1141] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1142] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1143] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1144] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1145] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1146] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1147] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1148] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1149] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1150] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1151] As an embodiment of the present invention, a system is constructed as follows.

[1152] System configuration

[1153] The system consists of three main components: a server, a terminal, and a user. The server is responsible for managing materials and performing search functions, and the terminal provides an interface for user input and display of results. Users use the terminal to search for materials.

[1154] Program processing

[1155] Study and storage of materials

[1156] The server trains the generative AI model on new materials and stores them in storage.

[1157] 1. Inputting materials

[1158] The server receives text data of documents as input, such as business reports or marketing strategies.

[1159] 2. Generate feature vectors

[1160] The server uses a generative AI model to generate a feature vector from the document, which is a numerical representation of the document's content, making it easier to analyze the document's characteristics.

[1161] 3. Storage in the database

[1162] The server stores the generated feature vectors and the documents themselves in a database, along with metadata about the documents (creation date, creator, etc.).

[1163] Receiving requests from users

[1164] The user uses a dedicated terminal to input the image and keywords of the material to be searched.

[1165] 1. Enter your request

[1166] The user inputs a search query (keywords, sentences, etc.) into the input form of the search terminal. For example, the user inputs "materials related to marketing strategies for 2023."

[1167] 2. Sending the request

[1168] The terminal transmits the query entered by the user to the server.

[1169] Search for documents

[1170] The server analyzes the user's request and searches the database for the most suitable material.

[1171] 1. Parsing the query

[1172] The server analyzes the received query using natural language processing technology and understands the meaning of the query.

[1173] 2. Generate a query feature vector

[1174] The server generates a feature vector from the query, which is a numerical representation of the query's meaning.

[1175] 3. Comparison of materials

[1176] The server compares the feature vectors of the generated query with the feature vectors of the documents stored in the database.

[1177] 4. Selection of the most suitable materials

[1178] The server uses a similarity calculation (for example, cosine similarity) to select materials that are highly similar to the query.

[1179] Material suggestions

[1180] The server transmits a list of the selected materials to the user's terminal.

[1181] 1. Creating a list of materials

[1182] The server sorts the materials in descending order of relevance and creates a list.

[1183] 2. Submission of materials

[1184] The server transmits the material list to the user's terminal.

[1185] 3. View the list

[1186] The terminal visually displays the list of received materials to the user, including the title, summary, and creation date of each material.

[1187] 4. Check the details

[1188] The user selects a document in the list and checks its detailed contents.

[1189] Specific examples

[1190] For example, if a company provides a document titled "Marketing Strategy 2023" as input to the server, the server will train this document using a generative AI model and generate a feature vector. The generated feature vector and the document itself will then be stored in the database.

[1191] Next, the user enters "materials related to marketing strategies for 2023" as a search query into the terminal and submits it. The server analyzes this query and generates a feature vector. After comparing it with materials in the database, "Marketing strategies for 2023" in particular shows a high degree of relevance, so this material is selected.

[1192] Finally, a list of selected materials is sent to the user's device, where the user can review it. By selecting "Marketing Strategy 2023" from the list and checking the details, the user can efficiently obtain the information they are looking for.

[1193] This system allows users to quickly find relevant materials, improving work efficiency.

[1194] The processing flow will be explained below.

[1195] Program processing

[1196] Study and storage of materials

[1197] Step 1:

[1198] The server receives as input the text data of the materials, such as a PDF file or a presentation file about a marketing strategy.

[1199] Step 2:

[1200] The server passes the received documents to a generative AI model to generate a feature vector, which is a numerical representation of the document's content.

[1201] Step 3:

[1202] The server stores the generated feature vectors and the documents themselves in a database, along with the document metadata (creation date, creator, etc.).

[1203] Receiving requests from users

[1204] Step 4:

[1205] The user uses a dedicated terminal to input a query into an input form to search for materials, for example, "materials related to marketing strategies for 2023."

[1206] Step 5:

[1207] The terminal sends the entered query to the server, which sends the query as text data.

[1208] Search for documents

[1209] Step 6:

[1210] The server receives the query sent by the user and analyzes it using natural language processing techniques to accurately understand the content and meaning of the query.

[1211] Step 7:

[1212] The server generates a feature vector from the query, which is a numerical representation of the query content.

[1213] Step 8:

[1214] The server compares the feature vectors of the documents stored in the database with the feature vectors of the query, for example, by calculating the cosine similarity to evaluate the similarity.

[1215] Step 9:

[1216] The server selects the most similar documents and, based on the comparison results, chooses the document that best matches the query.

[1217] Material suggestions

[1218] Step 10:

[1219] The server lists the selected materials and sorts them in order of relevance. This list includes the title, summary, and creation date of each proposal.

[1220] Step 11:

[1221] The server sends the created list of materials to the user's terminal as text data.

[1222] Step 12:

[1223] The terminal visually displays a list of received materials to the user, allowing the user to check the title, summary, creation date, etc. at a glance.

[1224] Step 13:

[1225] The user selects a document in the list and performs an action to check the details, for example, clicking on a specific document to go to its details page.

[1226] Specific examples

[1227] For example, when a company provides a document called "Marketing Strategy 2023" as input to the server, the following steps are executed:

[1228] Step 1:

[1229] The server receives text data of a document called "Marketing Strategy 2023."

[1230] Step 2:

[1231] The server passes the material to a generative AI model, which generates a feature vector for the material.

[1232] Step 3:

[1233] The server stores the generated feature vectors and data in a database.

[1234] Step 4-5:

[1235] The user enters the query "materials related to marketing strategy for 2023" into the terminal, which then sends it to the server.

[1236] Steps 6-9:

[1237] The server analyzes the query, generates a feature vector of the query, compares it with materials in the database, and selects "Marketing Strategy 2023" as the most suitable material.

[1238] Steps 10-13:

[1239] The server generates a list of materials and transmits it to the terminal, which displays it, and the user selects a material from the list and checks the details.

[1240] The above processing steps enable users to efficiently search for and obtain the materials they need.

[1241] Example 1

[1242] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1243] With conventional document management systems, it was difficult to efficiently search and retrieve the desired information from a huge amount of documents. Furthermore, when users searched, documents that were not relevant to the query they entered were often displayed, reducing the user's work efficiency. Furthermore, searches often did not fully utilize the document metadata, making it difficult to perform highly accurate searches.

[1244] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1245] In this invention, the server includes means for inputting materials, means for generating a feature vector from the materials, means for storing the generated feature vector in a database, means for receiving a search request from a user, means for generating a query feature vector from the search request, means for comparing the feature vector in the database with the feature vector of the query, means for selecting highly relevant materials based on the comparison results, means for generating and transmitting a list of the selected materials, and means for displaying the selected materials. This makes it possible to quickly and accurately search for and obtain materials highly relevant to the search query entered by the user, thereby improving the user's work efficiency.

[1246] "Materials" refers to documents and data that record information, including, for example, business reports and marketing strategies.

[1247] A "feature vector" is a numerical representation of the content of a document or query, making its features easier to analyze.

[1248] A "database" is a management system for storing materials and feature vectors, and includes, for example, a relational database.

[1249] "User" means any individual or organization that uses the System to search and view materials.

[1250] "Search request" refers to a query or keywords entered by a user to search for materials.

[1251] A "query" is a keyword or sentence that a user inputs as a search request, and includes information to be searched.

[1252] "Natural language processing technology" is a technology that enables computers to understand and analyze human language, and includes processes such as text analysis, semantic analysis, and context understanding.

[1253] The "degree of relevance" is an index showing the relevance between the query and the material, and is a numerical value obtained by similarity calculation or the like.

[1254] A "list" is a list of selected materials, and is displayed visually to the user.

[1255] "Metadata" refers to additional information about a material, including information such as the creation date and creator.

[1256] MODE FOR CARRYING OUT THE INVENTION

[1257] This invention is directed to a system that is composed of three main elements: a server, a terminal, and a user. A specific embodiment for implementing the program for this system will be described below.

[1258] System configuration

[1259] The server is responsible for managing materials and search functions, providing functions such as inputting materials, generating feature vectors, storing them in a database, processing user search requests, analyzing queries, comparing materials, selecting the most suitable materials, creating a material list, and sending the material list. The server generates material feature vectors using a generative AI model (e.g., OpenAI's GPT-3) and analyzes queries using natural language processing technology. It also uses a database (e.g., MySQL or PostgreSQL, which are relational databases) to store materials, feature vectors, and metadata.

[1260] The terminal provides an interface for user input and results display. The terminal receives the user's search request (query) and sends it to the server. The terminal also visually displays the list of materials received from the server to the user.

[1261] Users search for materials using a dedicated terminal. They input a search query through the terminal interface and check detailed materials from the list of materials presented as results.

[1262] Program processing explanation

[1263] The server receives the text data of the document and generates a feature vector from the document using a generative AI model (e.g., GPT-3). The generated feature vector is stored in a relational database, along with metadata about the document (such as creation date and author).

[1264] For example, if a company provides a document titled "Marketing Strategy 2023" to the server, the server will train the document using a generative AI model to generate a feature vector. This feature vector and the document itself will then be stored in the database.

[1265] A user inputs a search query using a dedicated terminal. For example, if a query such as "Marketing strategy materials for 2023" is input, the terminal sends the query to the server. The server then analyzes the received query using natural language processing technology and generates a feature vector to understand the meaning of the query.

[1266] The generated query feature vector is compared with the feature vectors of the documents in the database. This comparison uses a similarity calculation (e.g., cosine similarity). As a result, documents with high similarity to the query are selected.

[1267] The selected materials are then listed in descending order of relevance, and this list is sent to the user's device. The user's device visually displays the received materials list, allowing the user to select a material in the list and view its details.

[1268] Specific examples

[1269] For example, if a company provides a document titled "Marketing Strategy 2023" to the server, the server will train the document using a generative AI model to generate a feature vector. This feature vector and the document itself will then be stored in the database.

[1270] Next, the user enters "materials related to marketing strategies for 2023" as a search query into the terminal and submits it. The server analyzes this query and generates a feature vector. After comparing it with materials in the database, "Marketing strategies for 2023" in particular shows a high degree of relevance, so this material is selected.

[1271] Finally, a list of selected materials is sent to the user's device, where the user can review it. By selecting "Marketing Strategy 2023" from the list and checking the details, the user can efficiently obtain the information they are looking for.

[1272] Prompt Sentence Examples

[1273] "Find resources for 2023 marketing strategies."

[1274] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1275] System program processing flow

[1276] Step 1: Enter your materials

[1277] The server receives the text data of the document as input. For example, if a company provides a document called "Marketing Strategy 2023," the text data of this document is sent to the server. Specifically, the server receives the document in the form of an HTTP request or file upload and temporarily stores it in memory as input data.

[1278] Input: Text data of the document (e.g., "Marketing Strategy 2023")

[1279] Output: Data stored on the server

[1280] Step 2: Generate feature vectors

[1281] The server inputs the received material into a generative AI model (e.g., GPT-3) to generate a feature vector for the material. Specifically, the server passes the text of the material to the generative AI model, and the model extracts themes and keywords from the material to generate a feature vector.

[1282] Input: Text data of the material

[1283] Output: Feature vector of the material

[1284] Step 3: Store in the database

[1285] The server stores the generated feature vectors, the documents themselves, and metadata in a database. Specifically, the server uses SQL commands to save the document's feature vectors, text data, creation date and time, creator, and other metadata in a database (e.g., MySQL or PostgreSQL).

[1286] Input: feature vectors, text data, and metadata

[1287] Output: Material data and feature vectors stored in a database

[1288] Step 4: Entering Requests

[1289] A user inputs a search query using the interface of a dedicated terminal. For example, the user inputs the query "materials related to marketing strategies for 2023." Specifically, the user inputs the query into an input form using a web browser or a client application.

[1290] Input: Search query (e.g., "2023 marketing strategy materials")

[1291] Output: A search query is entered and persisted to the device.

[1292] Step 5: Submitting the request

[1293] The terminal sends the query entered by the user to the server. Specifically, the terminal sends an HTTP POST request and passes the query data to the server.

[1294] Input: search query

[1295] Output: Query data sent to the server

[1296] Step 6: Parsing the query

[1297] The server analyzes the received query using natural language processing technology to understand its meaning. Specifically, the server analyzes the query using a generative AI model to extract keywords and sentence structure.

[1298] Input: search query

[1299] Output: Query analysis results (extracted keywords, etc.)

[1300] Step 7: Generate a feature vector for the query

[1301] The server generates a feature vector from the query. Specifically, the server inputs the query text into a generative AI model and generates a feature vector that quantifies the meaning of the query.

[1302] Input: Query text data

[1303] Output: Query feature vector

[1304] Step 8: Compare materials

[1305] The server compares the feature vectors of the generated query with the feature vectors of the data stored in the database. Specifically, the server uses a similarity calculation algorithm (e.g., cosine similarity) to quickly compare the query and data vectors.

[1306] Input: feature vectors in the database, query feature vectors

[1307] Output: Similarity list between documents and queries

[1308] Step 9: Select the best materials

[1309] Based on the results of the similarity calculation, the server selects materials that have a high similarity to the query. Specifically, the server sorts the materials in descending order of similarity and selects the top-ranked materials.

[1310] Input: Similarity list between document and query

[1311] Output: List of highly relevant materials

[1312] Step 10: List your resources

[1313] The server sorts the documents in descending order of relevance and creates a list. Specifically, the server formats the selected documents in JSON format and prepares to return them as a response.

[1314] Input: Highly relevant materials

[1315] Output: Document list (JSON format)

[1316] Step 11: Submit your materials

[1317] The server sends the document list to the user's device. Specifically, the server generates an HTTP response and sends the document list to the device in JSON format.

[1318] Input: Document list (JSON format)

[1319] Output: List of materials sent to the user's terminal

[1320] Step 12: View the list

[1321] The device visually displays the list of received materials to the user, including the material's title, summary, creation date, etc. Specifically, the device parses the received JSON data and displays it on the user interface using HTML and CSS.

[1322] Input: List of received materials (JSON format)

[1323] Output: A visually displayed list of materials

[1324] Step 13: Check the details

[1325] The user selects a document from the list to view its details. When the user clicks on a document, the device opens a page displaying the details and then sends a request to the server to retrieve the full text of the document.

[1326] Input: User selection of material

[1327] Output: Detailed content of the document

[1328] (Application example 1)

[1329] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1330] Conventional search systems often have difficulty quickly and accurately finding the information and products users are looking for. In particular, virtual stores often contain a large amount of product information, making it difficult for users to easily find the products they are looking for. Furthermore, there is a lack of product suggestion systems that effectively utilize natural language processing technology, which means they are unable to accurately understand users' search intent and suggest appropriate products.

[1331] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1332] In this invention, the server includes means for inputting materials, means for generating a feature vector from the materials, means for storing the generated feature vector in a database, means for receiving a search request from a user, means for generating a query feature vector from the search request, means for comparing the feature vector in the database with the feature vector of the query, means for selecting highly relevant materials based on the comparison results, means for displaying the selected materials, means for receiving a product search request from a user using a smart device and analyzing the search request, means for quantifying the query feature vector using a generative AI model, and means for comparing the query feature vector with the feature vector in the product database by similarity calculation and suggesting highly relevant products. This enables the user to quickly and accurately search for and suggest products they are looking for in a virtual store, significantly improving the user's shopping experience.

[1333] "Materials" means information or data that is collected, stored or managed for a specific purpose.

[1334] A "feature vector" is a numerical representation of the content of a document or query, making it easier to analyze and compare.

[1335] A "database" is a structured collection of information and a system for storing, managing, and retrieving data.

[1336] "User" means any person or entity that utilizes the System to enter search requests and obtain results.

[1337] A "generative AI model" is a model that uses artificial intelligence technology to generate feature vectors from materials or queries.

[1338] A "smart device" is an electronic device that can connect to the Internet and has multiple functions (e.g., a smartphone or a head-mounted display).

[1339] "Natural language processing technology" is a technology that uses computers to understand and process human language (natural language).

[1340] A "search request" is an instruction or query for searching information that a user inputs to a system.

[1341] A "query" is a keyword or phrase that a user enters into a system to search for a specific piece of information.

[1342] "Similarity calculation" is a method for calculating the similarity between the feature vector of a query and the feature vectors in a database.

[1343] "Products" are objects for sale that are searched and suggested within the virtual store.

[1344] A "suggestion system" is a system that selects and displays appropriate materials and products based on a user's search request.

[1345] "Relevance" is an indicator of how closely a query matches materials or products in a database.

[1346] As an embodiment of the present invention, a system is constructed as follows. The system is composed of the main elements of a server, a terminal, and a user. The server is responsible for managing materials and products and for search functions, and the terminal provides an interface for user input and display of results. Users use the terminal to search for materials and products.

[1347] System configuration

[1348] server

[1349] The server has the following features:

[1350] 1. Input of materials and products

[1351] The server receives text data of materials and products (e.g., product names, descriptions, attributes, prices, etc.) as input.

[1352] 2. Generate feature vectors

[1353] The server generates feature vectors from documents and products using a generative AI model (e.g., a model using TensorFlow or PyTorch).

[1354] 3. Storage in the database

[1355] The server stores the generated feature vectors and the materials and products themselves in a database, along with related metadata (such as registration date and time, category, etc.).

[1356] 4. Processing Search Requests

[1357] The server receives a search request from a user, analyzes the query using natural language processing techniques (e.g., spaCy or NLTK), and generates a corresponding feature vector.

[1358] 5. Similarity calculation

[1359] The server compares the feature vectors of the generated query with the feature vectors in the database using similarity calculations (for example, cosine similarity calculations using Scikit-learn) and selects materials and products with high matching scores.

[1360] 6. Submitting the results

[1361] The server transmits a list of the selected materials and products to the user's terminal and displays it visually.

[1362] Terminal

[1363] The terminal has the following features:

[1364] 1. Accepting input from the user

[1365] The terminal receives the user's search query (for example, "blue summer dress") from an input form and sends it to the server.

[1366] 2. Displaying the results

[1367] The terminal visually displays the search results received from the server to the user, including the product title, summary, price, and image.

[1368] User

[1369] A user uses the system in the following steps:

[1370] 1. Enter your search query

[1371] A user inputs a search query using a smart device (e.g., a smartphone or a head-mounted display).

[1372] 2. Check the search results

[1373] The user checks the search results displayed on the terminal, selects the required materials or products, and checks the details.

[1374] Specific examples

[1375] For example, when a user uses a smartphone to enter a search query such as "I want a blue summer dress," the device sends this query to a server. The server analyzes the query and generates a feature vector using a generative AI model. It compares this with the feature vectors of products stored in a database and displays the most suitable products (for example, "A lightweight, breathable blue summer dress") in a list ordered by relevance to the user. The user can then select a product from the results and view detailed information.

[1376] Prompt Sentence Examples

[1377] An example prompt for a generative AI model is:

[1378] Generate a feature vector based on the following product information: product name 'Blue summer dress', description 'Lightweight and breathable', attributes 'Color: Blue Season: Summer', price '3000 yen'.

[1379] In this way, users can quickly and accurately search for the materials and products they need and efficiently obtain the information they desire.

[1380] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1381] Step 1:

[1382] The server receives text data of materials and products as input. A user or administrator sends information about the materials and products (e.g., product name, description, attributes, price, etc.) to the server via an input device. This input data is in text format and includes a detailed description of the materials and products. The server prepares the received data to be passed to the generative AI model.

[1383] Step 2:

[1384] The server generates a feature vector from the received text data of the documents or products. Using a generative AI model (for example, a model using TensorFlow or PyTorch), the text data is converted into a feature vector. This feature vector is a numerical representation of the content of the documents or products, and is used for searching and comparison.

[1385] Step 3:

[1386] The server stores the generated feature vectors and the data of the materials and products themselves in a database. Related metadata (registration date and time, category, etc.) is also saved. This allows for quick retrieval of data in response to future search requests.

[1387] Step 4:

[1388] A user inputs a search query using a smart device (e.g., a smartphone or a head-mounted display). For example, the user inputs a query such as "I want a blue summer dress" into an input form. The device then sends this query to the server.

[1389] Step 5:

[1390] The server receives a search query from a user and analyzes it using natural language processing techniques (e.g., spaCy or NLTK). It extracts important keywords and phrases from the query and generates a corresponding feature vector, which helps understand the user's intent.

[1391] Step 6:

[1392] The server compares the feature vector of the generated query with the feature vectors in the database. It calculates the similarity between the query and the feature vectors of the materials or products using methods such as cosine similarity calculations using Scikit-learn. Based on this similarity, it selects materials or products with the highest matching scores.

[1393] Step 7:

[1394] The server generates a list of selected materials and products, sorts them by relevance, and sends the list to the user's device. The list includes product titles, summaries, prices, images, etc.

[1395] Step 8:

[1396] The terminal visually displays the list of materials and products received from the server to the user. The user can view the results and check detailed information as needed. Specifically, the user uses an interface that displays the product name, description, price, image, etc.

[1397] Step 9:

[1398] The user can select the desired product from the displayed list and check the detailed information. For example, they can view the detailed information of a "lightweight and breathable blue summer dress" and consider purchasing it.

[1399] Through the above processing steps, the user can quickly and accurately search for the desired materials or products and efficiently obtain the desired information.

[1400] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1401] As an embodiment of this invention, a system that combines document management and user emotion recognition is constructed. The system is composed of a server, a terminal, a user, and an emotion engine. The server is responsible for document management and search functions, while the terminal provides an interface for user input and result display. The user uses the terminal to search for documents, and the emotion engine is responsible for recognizing the user's emotions.

[1402] Program processing

[1403] Study and storage of materials

[1404] The server trains the generative AI model on new materials and stores them in storage.

[1405] 1. Inputting materials

[1406] The server receives text data of materials as input, such as PDF files or presentation files about marketing strategies.

[1407] 2. Generate feature vectors

[1408] The server uses a generative AI model to generate a feature vector from the document, which is a numerical representation of the document's content.

[1409] 3. Storage in the database

[1410] The server stores the generated feature vectors and the documents themselves in a database, along with the document metadata (creation date, creator, etc.).

[1411] Receiving requests from users

[1412] The user uses a dedicated terminal to input the image and keywords of the material to be searched.

[1413] 1. Enter your request

[1414] The user inputs a search query (keywords, sentences, etc.) into the input form of the search terminal. For example, the user inputs "materials related to marketing strategies for 2023."

[1415] 2. Sending the request

[1416] The terminal sends the entered query to the server, which sends the query as text data.

[1417] Emotion recognition

[1418] To recognize the user's emotions, the system is equipped with an emotion engine.

[1419] 1. Acquiring Emotion Data

[1420] When a user enters a query, the device captures the user's facial expressions and voice in real time through a camera, microphone, etc.

[1421] 2. Emotion Analysis

[1422] The server uses an emotion engine to analyze the captured data and recognize the user's emotions (e.g., joy, sadness, surprise, impatience, etc.).

[1423] Search for documents

[1424] The server analyzes the user's request and searches the database for the most suitable material.

[1425] 1. Parsing the query

[1426] The server analyzes the received query using natural language processing technology and understands the meaning of the query.

[1427] 2. Generate a feature vector for the query

[1428] The server generates a feature vector from the query, which is a numerical representation of the query content.

[1429] 3. Comparison of materials

[1430] The server compares the feature vectors of the documents stored in the database with the feature vectors of the query, for example, by calculating the cosine similarity to evaluate the similarity.

[1431] 4. Adjusting the fit

[1432] The server adjusts the selection criteria for materials based on the user's emotions recognized by the emotion engine. For example, if the user is in a hurry, the system will prioritize more intuitive and concise materials.

[1433] 5. Selection of the most suitable materials

[1434] The server selects materials with high similarity and creates an optimal list of materials that reflects the results of the emotion engine.

[1435] Material suggestions

[1436] The server transmits a list of the selected materials to the user's terminal.

[1437] 1. Creating a list of materials

[1438] The server sorts the documents in descending order of relevance and creates a list of them, including the title, summary, and creation date of each document.

[1439] 2. Submission of materials

[1440] The server sends the created list of materials to the user's terminal as text data.

[1441] 3. View the list

[1442] The terminal visually displays a list of received materials to the user, allowing the user to check the title, summary, creation date, etc. at a glance.

[1443] 4. Check the details

[1444] The user selects a document in the list and performs an action to check the details, for example, clicking on a specific document to go to its details page.

[1445] Specific examples

[1446] For example, when a company provides a document called "Marketing Strategy 2023" as input to the server, the following steps are executed:

[1447] Steps 1-3:

[1448] The server trains a generative AI model on this material, generates a feature vector for the material, and stores it in a database.

[1449] Step 4-5:

[1450] The user enters the query "materials related to marketing strategy for 2023" into the terminal, which then sends it to the server.

[1451] Steps 6-7:

[1452] The server analyzes the query, generates a feature vector for the query, and compares it with the material in the database.

[1453] Step 8:

[1454] The device captures the user's facial expressions and voice through a camera and microphone, and the server analyzes the user's emotions using an emotion engine. For example, if the user is feeling anxious, the system will adjust the materials it selects based on this information.

[1455] Steps 9-13:

[1456] The server selects highly relevant materials, including "Marketing Strategy 2023," generates a list of materials, and sends it to the user's device. The device displays the list, and the user can select a material from the list to check its details.

[1457] This system allows users to efficiently search and obtain the materials they need, and also allows them to receive optimal suggestions based on their emotions.

[1458] The processing flow will be explained below.

[1459] Program processing

[1460] Study and storage of materials

[1461] Step 1:

[1462] The server receives as input the text data of the document, for example a PDF file or a presentation file about a new marketing strategy.

[1463] Step 2:

[1464] The server passes the received documents to the generative AI model, which generates a feature vector from the documents. This feature vector is a numerical representation of the contents of the documents.

[1465] Step 3:

[1466] The server stores the generated feature vectors in a database, as well as the materials themselves and their metadata (creation date, creator, etc.).

[1467] Receiving requests from users

[1468] Step 4:

[1469] Users use a dedicated terminal to input images and keywords for the materials they are searching for, for example, "materials related to marketing strategies for 2023."

[1470] Step 5:

[1471] The terminal sends the entered query to the server, which sends the query as text data.

[1472] Emotion recognition

[1473] Step 6:

[1474] The device captures the user's facial expressions and voice data in real time through a camera and microphone while the user enters a query.

[1475] Step 7:

[1476] The server uses an emotion engine to analyze the captured data and recognize the user's emotions, for example, whether the user is anxious or calm.

[1477] Search for documents

[1478] Step 8:

[1479] The server analyzes the received query using natural language processing technology to understand its meaning, clarifying the meaning of keywords and phrases.

[1480] Step 9:

[1481] The server generates a feature vector from the query, which is a numerical representation of the query content.

[1482] Step 10:

[1483] The server compares the feature vectors in the database with the feature vectors of the query, for example by calculating cosine similarity to assess relevance.

[1484] Step 11:

[1485] The server adjusts the selection criteria for materials based on the user's emotions recognized by the emotion engine. For example, if the user is in a hurry, the server adjusts the selection criteria to prioritize intuitive and concise materials.

[1486] Step 12:

[1487] The server selects highly relevant materials and lists the most suitable materials based on the similarity between the query and the feature vectors of the materials.

[1488] Material suggestions

[1489] Step 13:

[1490] The server creates a list of the selected materials, sorted by relevance, including title, summary, and creation date.

[1491] Step 14:

[1492] The server sends the created list of materials to the user's terminal as text data.

[1493] Step 15:

[1494] The terminal visually displays the list of received materials to the user, including the title, summary, and creation date.

[1495] Step 16:

[1496] The user can select an item in the list to view its details, for example, by clicking on a specific item to go to its details page.

[1497] Specific examples

[1498] For example, when a company provides a document called "Marketing Strategy 2023" as input to the server, the following steps are executed:

[1499] Step 1:

[1500] The server receives text data of a document called "Marketing Strategy 2023."

[1501] Step 2:

[1502] The server passes the material to a generative AI model to generate a feature vector.

[1503] Step 3:

[1504] The server stores the generated feature vectors, the material itself, and its metadata in a database.

[1505] Step 4:

[1506] The user enters the query "Marketing strategy materials for 2023" into the terminal.

[1507] Step 5:

[1508] The terminal sends this query to the server.

[1509] Step 6:

[1510] The device uses a camera and microphone to capture the user's emotional data.

[1511] Step 7:

[1512] The server analyzes the user's emotions using an emotion engine and recognizes that the user is, for example, "anxious."

[1513] Steps 8-12:

[1514] The server analyzes the query, generates a feature vector, compares it with the database, and adjusts the selection criteria based on the user's sentiment to select the most suitable list of materials.

[1515] Steps 13-16:

[1516] The server generates a list of materials and sends it to the user's terminal, which displays it. The user selects a material and checks its details.

[1517] This system allows users to efficiently search and obtain the materials they need, and also allows them to receive optimal suggestions based on their emotions.

[1518] Example 2

[1519] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1520] Existing document management systems search and suggest documents without considering the user's emotions, making it difficult to intuitively provide the documents the user needs. In particular, the optimal documents may differ depending on the user's emotional state, which can lead to a poor user experience.

[1521] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for inputting materials, means for generating a feature vector from the materials, means for storing the generated feature vector in a database, means for receiving a search request from a user, means for generating a query feature vector from the search request, means for comparing the feature vector in the database with the feature vector of the query, means for selecting highly suitable materials based on the comparison results, means for displaying the selected materials, means for recognizing the user's emotions, and means for adjusting the material selection criteria based on the emotions. This makes it possible to search for and suggest optimal materials according to the user's emotional state.

[1522] "Materials" refers to documents or data in which information is organized in the form of text, images, audio, etc.

[1523] A "feature vector" is vector data that numerically represents the content of a document or the meaning of a query.

[1524] A "database" is a system that systematically stores and manages materials, feature vectors, metadata, etc.

[1525] A "user search request" refers to keywords or sentences that a user inputs to the system to search for specific materials.

[1526] A "query feature vector" is numerical vector data generated based on a search request.

[1527] The "comparison result" refers to the evaluation result of similarity or compatibility obtained by comparing the feature vectors in the database with the feature vectors of the query.

[1528] "Material selection criteria" refers to the criteria or indicators used to select materials, including sentiment and relevance.

[1529] "User's emotions" refers to the user's psychological or emotional state (for example, joy, sadness, surprise, impatience, etc.).

[1530] "Camera and microphone" refers to input devices for capturing the user's facial expressions and voice.

[1531] A "generative AI model" is a generative model that uses artificial intelligence and is an algorithm used to generate feature vectors from materials or queries.

[1532] MODE FOR CARRYING OUT THE INVENTION

[1533] As an embodiment of this invention, we will build a system that combines document management and user emotion recognition. The system is composed of a server, a terminal, a user, and an emotion engine. The server is responsible for document management and search functions, while the terminal provides an interface for user input and display of results. The user uses the terminal to search for documents, and the emotion engine is responsible for recognizing the user's emotions. Specific embodiments of each element of this system will be described below.

[1534] Study and storage of materials

[1535] The server trains new documents using a generative AI model. Specifically, the server receives the document's text data as input and generates a feature vector using a generative AI model (e.g., GPT-4). This feature vector is a numerical representation of the document's content and is stored in a database. The stored data includes not only the feature vector, but also the document itself and its metadata (creation date and time, creator, etc.).

[1536] Receiving requests from users

[1537] A user attempts to search for materials using a dedicated terminal. The user enters a search query (e.g., "materials related to marketing strategies for 2023") into the terminal's input form, and the terminal sends this query as text data to the server. The query is sent to the server as an HTTP request and processed by the server.

[1538] Emotion recognition

[1539] When a user enters a query, the device captures the user's facial expressions and voice in real time through a camera and microphone. This data is sent to a server, which then uses an emotion engine to analyze the user's emotions. For example, it uses Microsoft's Azure Face API or Google Cloud Natural Language emotion analysis tools to recognize the user's emotions (happiness, sadness, surprise, impatience, etc.).

[1540] Search for documents

[1541] The server analyzes the user's request and searches the database for the most appropriate materials. First, it analyzes the received query using natural language processing technology (e.g., BERT or GPT-4) to understand the meaning of the query. Then, it uses a generative AI model to generate a feature vector from the query. The server compares this feature vector with feature vectors in the database and evaluates the similarity, for example, by calculating cosine similarity. Furthermore, it adjusts the material selection criteria based on the user's emotions recognized by the emotion engine, and selects materials with high similarity.

[1542] Material suggestions

[1543] The server sends a list of selected materials to the user's device. The list is sorted in order of relevance and includes information such as the material's title, summary, and creation date. The device visually displays the received material list, and the user can select a material from the list to check its details. Specifically, clicking on the list on the device takes the user to the details page.

[1544] Specific examples

[1545] For example, if a company provides a document titled "Marketing Strategy 2023" as input, the server trains the document through a generative AI model, generates a feature vector, and stores it in a database. The user then enters a query, "Documents related to marketing strategy for 2023," and the device sends this to the server. The server analyzes the query, generates a specific feature vector, and compares it with documents in the database. The device captures the user's facial expressions and voice and analyzes their emotions using an emotion engine. For example, if the user is anxious, the system adjusts the documents it selects based on this information. Finally, the server selects documents with high relevance, including "Marketing Strategy 2023," generates a document list, and sends it to the device. The user can select a document from the list to view its details.

[1546] This system allows users to efficiently search and obtain the materials they need, and also allows them to receive optimal suggestions based on their emotions.

[1547] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1548] Step 1:

[1549] Inputting materials

[1550] The server receives text data of documents as input, such as PDF files or presentation files, which may be obtained from an internal management system or an external data source.

[1551] Input: Text data of the material (e.g., PDF file, Word file)

[1552] What it does: The server reads the document and extracts its contents as a string.

[1553] Output: Text data

[1554] Step 2:

[1555] Feature Vector Generation

[1556] The server uses a generative AI model (e.g., GPT-4) to generate a feature vector from the document. The feature vector is a numerical representation of the content of the document.

[1557] Input: Extracted text data

[1558] How it works: Text data is fed into a generative AI model to generate a feature vector.

[1559] Output: Feature vector

[1560] Step 3:

[1561] Storage in the database

[1562] The server stores the generated feature vectors and the materials themselves in a database together with metadata.

[1563] Input: feature vector, text data of the material, metadata (creation date, creator, etc.)

[1564] Operation: Issues an INSERT statement to the database and saves the data.

[1565] Output: Data is stored in the database

[1566] Step 4:

[1567] Entering a Request

[1568] The user enters a search query (for example, "materials related to marketing strategies for 2023") into the input form on the dedicated terminal.

[1569] Input: User's search query (keywords, sentences)

[1570] Action: Enter text using the keyboard on the device and click the send button.

[1571] Output: Text data of the search query

[1572] Step 5:

[1573] Submitting a request

[1574] The device sends the entered query to the server, sending data using an HTTP request.

[1575] Input: Text data of the search query

[1576] What it does: Sends a query to the server via an HTTP POST request.

[1577] Output: Query data sent to the server

[1578] Step 6:

[1579] Acquiring emotion data

[1580] When a user enters a query, the device captures the user's facial expressions and voice in real time through the camera and microphone.

[1581] Input: User's facial expression data, voice data

[1582] What it does: Captures a picture of your face with a webcam and records audio with a microphone.

[1583] Output: Captured facial expression data, audio data

[1584] Step 7:

[1585] Emotion Analysis

[1586] The server uses an emotion engine (e.g., Azure Face API) to analyze the captured data and recognize the user's emotions.

[1587] Input: Captured facial expression data, voice data

[1588] Operation: Facial images and voice data are input into the emotion engine to obtain emotional information.

[1589] Output: Emotion information (happiness, sadness, surprise, impatience, etc.)

[1590] Step 8:

[1591] Query Analysis

[1592] The server analyzes the received query using natural language processing technology (e.g., BERT or GPT-4) to understand the meaning of the query.

[1593] Input: Text data of the search query

[1594] How it works: Inputs query text into a natural language processing engine to perform analysis.

[1595] Output: Semantic information of the parsed query

[1596] Step 9:

[1597] Generate a query feature vector

[1598] The server generates a feature vector from the query.

[1599] Input: Parsed query semantics

[1600] How it works: Query information is input into the generative AI model, which generates a feature vector.

[1601] Output: Feature vector of the query

[1602] Step 10:

[1603] Comparison of materials

[1604] The server compares the feature vectors of the documents stored in the database with the feature vectors of the query, for example, by calculating the cosine similarity to evaluate the similarity.

[1605] Input: Query feature vector, database feature vector

[1606] Operation: Performs a cosine similarity calculation on each feature vector.

[1607] Output: Similarity score with each document

[1608] Step 11:

[1609] Adjusting the fit

[1610] The server adjusts the selection criteria for materials based on the user's emotions recognized by the emotion engine. If the user is in a hurry, intuitive and concise materials will be suggested first.

[1611] Input: Similarity score, emotion information

[1612] How it works: Adjust filtering conditions and score weighting based on emotional information.

[1613] Output: Adjusted similarity score

[1614] Step 12:

[1615] Selection of the most suitable materials

[1616] The server selects materials with high similarity and creates an optimal list of materials that reflects the results of the emotion engine.

[1617] Input: Adjusted similarity score

[1618] Operation: Select high-scoring materials and create a list.

[1619] Output: List of best materials

[1620] Step 13:

[1621] Creating a list of materials

[1622] The server sorts the documents in descending order of relevance and creates a list, including the document title, summary, creation date, etc.

[1623] Input: List of optimal materials

[1624] What it does: Sorts the list of documents and adds titles, summaries, creation dates, etc.

[1625] Output: Document list

[1626] Step 14:

[1627] Sending materials

[1628] The server sends the created document list to the user's terminal, and sends text data as an HTTP response.

[1629] Input: Document list

[1630] Operation: Returns the list of materials as an HTTP response in JSON format or similar.

[1631] Output: List of materials sent to the terminal

[1632] Step 15:

[1633] Viewing the list

[1634] The terminal visually displays a list of received materials to the user, allowing the user to check the title, summary, creation date, etc. at a glance.

[1635] Input: Document list

[1636] What it does: Renders a list visually with HTML and CSS.

[1637] Output: List of displayed materials

[1638] Step 16:

[1639] Check details

[1640] The user selects a document in the list and performs an action to check the details, for example, clicking on a specific document to go to its details page.

[1641] Input: Selection action on the material list

[1642] Action: Click on the document link to display the detailed view.

[1643] Output: Show detailed information

[1644] (Application example 2)

[1645] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1646] Conventional document management systems have the problem that when a user searches, the optimal document is not presented based on the user's emotions, making it difficult to effectively retrieve documents depending on the user's emotional state. Furthermore, search results may not fully satisfy the user's needs, resulting in a poor user experience. This is particularly important in virtual store environments, where users are required to quickly and accurately obtain products and information that match their emotions.

[1647] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting materials, means for generating a feature vector from the materials, means for storing the generated feature vector in a database, means for receiving a search request from a user, means for generating a query feature vector from the search request, means for comparing the feature vector in the database with the feature vector of the query, means for selecting materials with high suitability based on the comparison results, means for displaying the selected materials, means for recognizing the user's emotions, and means for adjusting the suitability based on the user's emotions. This makes it possible to quickly search for and suggest optimal materials and products that match the user's emotions, thereby improving the user experience.

[1648] "Materials" refers to information or data that users search or view.

[1649] A "feature vector" is vector-format data that quantifies the content of a document or query and characterizes that content.

[1650] A "database" is a collection of stored data that stores feature vectors and information for later retrieval and comparison.

[1651] A "search request" is a query or keyword entered by a user to search for a particular resource.

[1652] A "query" is a character string or a sentence in a natural language that a user uses to make a search request.

[1653] "Emotion" refers to a psychological state that can be recognized from a user's facial expression or voice, and includes states such as joy, sadness, surprise, and impatience.

[1654] "Relevance" is an index that indicates the relevance or match of a document to a user's search request or emotional state.

[1655] A "server" is a central computer that manages materials, performs searches, performs sentiment analysis, etc., and operates the entire system in conjunction with terminals.

[1656] A "terminal" is a device that provides an interface for users to search for and view materials.

[1657] An "emotion engine" is software or hardware that analyzes a user's emotions and recognizes emotions from facial expressions, voice, gestures, etc.

[1658] As an embodiment of the present invention, a system that combines document management and user emotion recognition is constructed. This system is composed of the following elements: a server, a terminal, a user, and an emotion engine.

[1659] 1. Server

[1660] The server is responsible for document management and search functions. It performs a series of processes, including inputting new documents, generating feature vectors, storing them in the database, and analyzing search requests. Specifically, it receives the document's text data, generates feature vectors using a generative AI model, and stores these in the database. It also receives search queries from users, analyzes them using natural language processing technology, and generates a query feature vector. It then compares the query feature vector with the feature vectors in the database to select documents with a high degree of relevance. During this process, the relevance is adjusted based on the user's emotions recognized by the emotion engine. By adjusting the relevance, it selects documents that best fit the user's emotional state.

[1661] 2. Terminal

[1662] The terminal provides an interface for users to input materials and make search requests. Users use the terminal to search for materials and display search results. The terminal captures the user's facial expressions and voice in real time through a camera and microphone and sends them to the server. When a user searches for materials, they enter a search query into the terminal's input form, which is sent to the server. The terminal visually displays the search results sent from the server, and the user can select the appropriate material from the list.

[1663] 3. Emotion Engine

[1664] The emotion engine recognizes the user's emotions. Specifically, when a user uses a device to input a query, it analyzes data captured through the device's camera and microphone to recognize the user's emotions. This allows the system to understand the user's emotional state. For example, if the user is in a hurry, the system will prioritize suggesting concise materials that can be understood immediately.

[1665] Specific examples

[1666] For example, in a VR shopping application for a virtual store, a user might say, "I'm looking for new sneakers." In this case, the device captures the user's facial expressions and voice in real time and sends them to the server. The server uses an emotion engine to analyze the user's emotions, and if it determines that the user is feeling anxious, it adjusts the suitability based on that information. As a result, sneakers that are most suitable for the user (popular items, highly reviewed items, etc.) are displayed preferentially.

[1667] Example prompts for generative AI models

[1668] "We suggest the best products for users who are in a hurry to find new sneakers."

[1669] This system allows users to efficiently search for and obtain the products they need in a VR environment and receive optimal suggestions based on their emotions at the time. The required hardware includes a VR headset, camera, and microphone. A typical VR headset (e.g., Meta Quest 2 or HTC Vive) is used as the specific device. Emotion analysis uses libraries such as Hume AI and Affectiva as emotion engines. On the server side, natural language processing technology (e.g., SpaCy and BERT) is used for document management and search.

[1670] In this way, the system of the invention can quickly search for and suggest the most suitable materials and products according to the user's emotional state, improving the user experience.

[1671] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1672] Step 1:

[1673] The user wears a VR headset and accesses a virtual store. The device captures the user's voice input, gaze, and facial expressions. As input, it acquires the user's voice query (e.g., "I'm looking for new sneakers") and emotional data including facial expressions and vocal tone.

[1674] Step 2:

[1675] The device extracts a search query from the voice input and sends it to the emotion engine. Specifically, it uses a microphone to convert the voice into text, which is then used as the query. The query is then sent to the server as text data.

[1676] Step 3:

[1677] The server analyzes the input query using natural language processing techniques (e.g., SpaCy, BERT) and generates a feature vector for the query. It receives the text data of the query as input and outputs the feature vector.

[1678] Step 4:

[1679] The device captures emotion data from the user's facial expressions and voice in real time and sends it to the emotion engine. Specifically, it uses a camera and microphone to capture the user's facial expressions and voice and sends the data to the server. The emotion data is output as extracted features.

[1680] Step 5:

[1681] The server analyzes the emotion data acquired by the emotion engine and recognizes the user's emotional state. It receives emotion data as input and outputs an emotional state such as joy, sadness, surprise, or impatience.

[1682] Step 6:

[1683] The server compares the query feature vector with the feature vectors in the database and calculates the similarity. Specifically, it uses mathematical techniques such as cosine similarity to compare the feature vectors of each document, and outputs the most suitable document from the database.

[1684] Step 7:

[1685] The server adjusts the relevance of search results based on the user's emotional state. For example, if the user is in a hurry, it will prioritize concise and intuitive materials. It receives the user's emotional state and the similarity of the materials as input, and outputs a list of materials with adjusted relevance.

[1686] Step 8:

[1687] The server selects the most relevant materials, creates a list of materials, and sends it to the user's terminal. The server receives the adjusted list of materials as input and sends it to the terminal as text data.

[1688] Step 9:

[1689] The device visually displays the received list of materials to the user. Specifically, the list is displayed on a display in the VR environment, and the user can select the appropriate material from the list. The output shows the title and summary of the material.

[1690] Step 10:

[1691] The user selects the document for which they wish to view details from the displayed list of documents. They are then taken to the details page of the selected document and can view the contents of the document. The system receives the user's selection as input and displays the detailed contents of the document as output.

[1692] Through this series of steps, users can efficiently search for and acquire the products they need in a VR environment and receive optimal suggestions based on their emotions at the time.

[1693] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1694] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1695] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1696] [Fourth embodiment]

[1697] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1698] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1699] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1700] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1701] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1702] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1703] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1704] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1705] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1706] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1707] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1708] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1709] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1710] As an embodiment of the present invention, a system is constructed as follows.

[1711] System configuration

[1712] The system consists of three main components: a server, a terminal, and a user. The server is responsible for managing materials and performing search functions, and the terminal provides an interface for user input and display of results. Users use the terminal to search for materials.

[1713] Program processing

[1714] Study and storage of materials

[1715] The server trains the generative AI model on new materials and stores them in storage.

[1716] 1. Inputting materials

[1717] The server receives text data of documents as input, such as business reports or marketing strategies.

[1718] 2. Generate feature vectors

[1719] The server uses a generative AI model to generate a feature vector from the document, which is a numerical representation of the document's content, making it easier to analyze the document's characteristics.

[1720] 3. Storage in the database

[1721] The server stores the generated feature vectors and the documents themselves in a database, along with metadata about the documents (creation date, creator, etc.).

[1722] Receiving requests from users

[1723] The user uses a dedicated terminal to input the image and keywords of the material to be searched.

[1724] 1. Enter your request

[1725] The user inputs a search query (keywords, sentences, etc.) into the input form of the search terminal. For example, the user inputs "materials related to marketing strategies for 2023."

[1726] 2. Sending the request

[1727] The terminal transmits the query entered by the user to the server.

[1728] Search for documents

[1729] The server analyzes the user's request and searches the database for the most suitable material.

[1730] 1. Parsing the query

[1731] The server analyzes the received query using natural language processing technology and understands the meaning of the query.

[1732] 2. Generate a feature vector for the query

[1733] The server generates a feature vector from the query, which is a numerical representation of the query's meaning.

[1734] 3. Comparison of materials

[1735] The server compares the feature vectors of the generated query with the feature vectors of the documents stored in the database.

[1736] 4. Selection of the most suitable materials

[1737] The server uses a similarity calculation (for example, cosine similarity) to select materials that are highly similar to the query.

[1738] Material suggestions

[1739] The server transmits a list of the selected materials to the user's terminal.

[1740] 1. Creating a list of materials

[1741] The server sorts the materials in descending order of relevance and creates a list.

[1742] 2. Submission of materials

[1743] The server transmits the material list to the user's terminal.

[1744] 3. View the list

[1745] The terminal visually displays the list of received materials to the user, including the title, summary, and creation date of each material.

[1746] 4. Check the details

[1747] The user selects a document in the list and checks its detailed contents.

[1748] Specific examples

[1749] For example, if a company provides a document titled "Marketing Strategy 2023" as input to the server, the server will train this document using a generative AI model and generate a feature vector. The generated feature vector and the document itself will then be stored in the database.

[1750] Next, the user enters "materials related to marketing strategies for 2023" as a search query into the terminal and submits it. The server analyzes this query and generates a feature vector. After comparing it with materials in the database, "Marketing strategies for 2023" in particular shows a high degree of relevance, so this material is selected.

[1751] Finally, a list of selected materials is sent to the user's device, where the user can review it. By selecting "Marketing Strategy 2023" from the list and checking the details, the user can efficiently obtain the information they are looking for.

[1752] This system allows users to quickly find relevant materials, improving work efficiency.

[1753] The processing flow will be explained below.

[1754] Program processing

[1755] Study and storage of materials

[1756] Step 1:

[1757] The server receives as input the text data of the materials, such as a PDF file or a presentation file about a marketing strategy.

[1758] Step 2:

[1759] The server passes the received documents to a generative AI model to generate a feature vector, which is a numerical representation of the document's content.

[1760] Step 3:

[1761] The server stores the generated feature vectors and the documents themselves in a database, along with the document metadata (creation date, creator, etc.).

[1762] Receiving requests from users

[1763] Step 4:

[1764] The user uses a dedicated terminal to input a query into an input form to search for materials, for example, "materials related to marketing strategies for 2023."

[1765] Step 5:

[1766] The terminal sends the entered query to the server, which sends the query as text data.

[1767] Search for documents

[1768] Step 6:

[1769] The server receives the query sent by the user and analyzes it using natural language processing techniques to accurately understand the content and meaning of the query.

[1770] Step 7:

[1771] The server generates a feature vector from the query, which is a numerical representation of the query content.

[1772] Step 8:

[1773] The server compares the feature vectors of the documents stored in the database with the feature vectors of the query, for example, by calculating the cosine similarity to evaluate the similarity.

[1774] Step 9:

[1775] The server selects the most similar documents and, based on the comparison results, chooses the document that best matches the query.

[1776] Material suggestions

[1777] Step 10:

[1778] The server lists the selected materials and sorts them in order of relevance. This list includes the title, summary, and creation date of each proposal.

[1779] Step 11:

[1780] The server sends the created list of materials to the user's terminal as text data.

[1781] Step 12:

[1782] The terminal visually displays a list of received materials to the user, allowing the user to check the title, summary, creation date, etc. at a glance.

[1783] Step 13:

[1784] The user selects a document in the list and performs an action to check the details, for example, clicking on a specific document to go to its details page.

[1785] Specific examples

[1786] For example, when a company provides a document called "Marketing Strategy 2023" as input to the server, the following steps are executed:

[1787] Step 1:

[1788] The server receives text data of a document called "Marketing Strategy 2023."

[1789] Step 2:

[1790] The server passes the material to a generative AI model, which generates a feature vector for the material.

[1791] Step 3:

[1792] The server stores the generated feature vectors and data in a database.

[1793] Step 4-5:

[1794] The user enters the query "materials related to marketing strategy for 2023" into the terminal, which then sends it to the server.

[1795] Steps 6-9:

[1796] The server analyzes the query, generates a feature vector of the query, compares it with materials in the database, and selects "Marketing Strategy 2023" as the most suitable material.

[1797] Steps 10-13:

[1798] The server generates a list of materials and transmits it to the terminal, which displays it, and the user selects a material from the list and checks the details.

[1799] The above processing steps enable users to efficiently search for and obtain the materials they need.

[1800] Example 1

[1801] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1802] With conventional document management systems, it was difficult to efficiently search and retrieve the desired information from a huge amount of documents. Furthermore, when users searched, documents that were not relevant to the query they entered were often displayed, reducing the user's work efficiency. Furthermore, searches often did not fully utilize the document metadata, making it difficult to perform highly accurate searches.

[1803] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1804] In this invention, the server includes means for inputting materials, means for generating a feature vector from the materials, means for storing the generated feature vector in a database, means for receiving a search request from a user, means for generating a query feature vector from the search request, means for comparing the feature vector in the database with the feature vector of the query, means for selecting highly relevant materials based on the comparison results, means for generating and transmitting a list of the selected materials, and means for displaying the selected materials. This makes it possible to quickly and accurately search for and obtain materials highly relevant to the search query entered by the user, thereby improving the user's work efficiency.

[1805] "Materials" refers to documents and data that record information, including, for example, business reports and marketing strategies.

[1806] A "feature vector" is a numerical representation of the content of a document or query, making its features easier to analyze.

[1807] A "database" is a management system for storing materials and feature vectors, and includes, for example, a relational database.

[1808] "User" means any individual or organization that uses the System to search and view materials.

[1809] "Search request" refers to a query or keywords entered by a user to search for materials.

[1810] A "query" is a keyword or sentence that a user inputs as a search request, and includes information to be searched.

[1811] "Natural language processing technology" is a technology that enables computers to understand and analyze human language, and includes processes such as text analysis, semantic analysis, and context understanding.

[1812] The "degree of relevance" is an index showing the relevance between the query and the material, and is a numerical value obtained by similarity calculation or the like.

[1813] A "list" is a list of selected materials, and is displayed visually to the user.

[1814] "Metadata" refers to additional information about a material, including information such as the creation date and creator.

[1815] MODE FOR CARRYING OUT THE INVENTION

[1816] This invention is directed to a system that is composed of three main elements: a server, a terminal, and a user. A specific embodiment for implementing the program for this system will be described below.

[1817] System configuration

[1818] The server is responsible for managing materials and search functions, providing functions such as inputting materials, generating feature vectors, storing them in a database, processing user search requests, analyzing queries, comparing materials, selecting the most suitable materials, creating a material list, and sending the material list. The server generates material feature vectors using a generative AI model (e.g., OpenAI's GPT-3) and analyzes queries using natural language processing technology. It also uses a database (e.g., MySQL or PostgreSQL, which are relational databases) to store materials, feature vectors, and metadata.

[1819] The terminal provides an interface for user input and results display. The terminal receives the user's search request (query) and sends it to the server. The terminal also visually displays the list of materials received from the server to the user.

[1820] Users search for materials using a dedicated terminal. They input a search query through the terminal interface and check detailed materials from the list of materials presented as results.

[1821] Program processing explanation

[1822] The server receives the text data of the document and generates a feature vector from the document using a generative AI model (e.g., GPT-3). The generated feature vector is stored in a relational database, along with metadata about the document (such as creation date and author).

[1823] For example, if a company provides a document titled "Marketing Strategy 2023" to the server, the server will train the document using a generative AI model to generate a feature vector. This feature vector and the document itself will then be stored in the database.

[1824] A user inputs a search query using a dedicated terminal. For example, if a query such as "Marketing strategy materials for 2023" is input, the terminal sends the query to the server. The server then analyzes the received query using natural language processing technology and generates a feature vector to understand the meaning of the query.

[1825] The generated query feature vector is compared with the feature vectors of the documents in the database. This comparison uses a similarity calculation (e.g., cosine similarity). As a result, documents with high similarity to the query are selected.

[1826] The selected materials are then listed in descending order of relevance, and this list is sent to the user's device. The user's device visually displays the received materials list, allowing the user to select a material in the list and view its details.

[1827] Specific examples

[1828] For example, if a company provides a document titled "Marketing Strategy 2023" to the server, the server will train the document using a generative AI model to generate a feature vector. This feature vector and the document itself will then be stored in the database.

[1829] Next, the user enters "materials related to marketing strategies for 2023" as a search query into the terminal and submits it. The server analyzes this query and generates a feature vector. After comparing it with materials in the database, "Marketing strategies for 2023" in particular shows a high degree of relevance, so this material is selected.

[1830] Finally, a list of selected materials is sent to the user's device, where the user can review it. By selecting "Marketing Strategy 2023" from the list and checking the details, the user can efficiently obtain the information they are looking for.

[1831] Prompt Sentence Examples

[1832] "Find resources for 2023 marketing strategies."

[1833] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1834] System program processing flow

[1835] Step 1: Enter your materials

[1836] The server receives the text data of the document as input. For example, if a company provides a document called "Marketing Strategy 2023," the text data of this document is sent to the server. Specifically, the server receives the document in the form of an HTTP request or file upload and temporarily stores it in memory as input data.

[1837] Input: Text data of the document (e.g., "Marketing Strategy 2023")

[1838] Output: Data stored on the server

[1839] Step 2: Generate feature vectors

[1840] The server inputs the received material into a generative AI model (e.g., GPT-3) to generate a feature vector for the material. Specifically, the server passes the text of the material to the generative AI model, and the model extracts themes and keywords from the material to generate a feature vector.

[1841] Input: Text data of the material

[1842] Output: Feature vector of the material

[1843] Step 3: Store in the database

[1844] The server stores the generated feature vectors, the documents themselves, and metadata in a database. Specifically, the server uses SQL commands to save the document's feature vectors, text data, creation date and time, creator, and other metadata in a database (e.g., MySQL or PostgreSQL).

[1845] Input: feature vectors, text data, and metadata

[1846] Output: Material data and feature vectors stored in a database

[1847] Step 4: Entering Requests

[1848] A user inputs a search query using the interface of a dedicated terminal. For example, the user inputs the query "materials related to marketing strategies for 2023." Specifically, the user inputs the query into an input form using a web browser or a client application.

[1849] Input: Search query (e.g., "2023 marketing strategy materials")

[1850] Output: A search query is entered and persisted to the device.

[1851] Step 5: Submitting the request

[1852] The terminal sends the query entered by the user to the server. Specifically, the terminal sends an HTTP POST request and passes the query data to the server.

[1853] Input: search query

[1854] Output: Query data sent to the server

[1855] Step 6: Parsing the query

[1856] The server analyzes the received query using natural language processing technology to understand its meaning. Specifically, the server analyzes the query using a generative AI model to extract keywords and sentence structure.

[1857] Input: search query

[1858] Output: Query analysis results (extracted keywords, etc.)

[1859] Step 7: Generate a feature vector for the query

[1860] The server generates a feature vector from the query. Specifically, the server inputs the query text into a generative AI model and generates a feature vector that quantifies the meaning of the query.

[1861] Input: Query text data

[1862] Output: Query feature vector

[1863] Step 8: Compare materials

[1864] The server compares the feature vectors of the generated query with the feature vectors of the data stored in the database. Specifically, the server uses a similarity calculation algorithm (e.g., cosine similarity) to quickly compare the query and data vectors.

[1865] Input: feature vectors in the database, query feature vectors

[1866] Output: Similarity list between documents and queries

[1867] Step 9: Select the best materials

[1868] Based on the results of the similarity calculation, the server selects materials that have a high similarity to the query. Specifically, the server sorts the materials in descending order of similarity and selects the top-ranked materials.

[1869] Input: Similarity list between document and query

[1870] Output: List of highly relevant materials

[1871] Step 10: List your resources

[1872] The server sorts the documents in descending order of relevance and creates a list. Specifically, the server formats the selected documents in JSON format and prepares to return them as a response.

[1873] Input: Highly relevant materials

[1874] Output: Document list (JSON format)

[1875] Step 11: Submit your materials

[1876] The server sends the document list to the user's device. Specifically, the server generates an HTTP response and sends the document list to the device in JSON format.

[1877] Input: Document list (JSON format)

[1878] Output: List of materials sent to the user's terminal

[1879] Step 12: View the list

[1880] The device visually displays the list of received materials to the user, including the material's title, summary, creation date, etc. Specifically, the device parses the received JSON data and displays it on the user interface using HTML and CSS.

[1881] Input: List of received materials (JSON format)

[1882] Output: A visually displayed list of materials

[1883] Step 13: Check the details

[1884] The user selects a document from the list to view its details. When the user clicks on a document, the device opens a page displaying the details and then sends a request to the server to retrieve the full text of the document.

[1885] Input: User selection of material

[1886] Output: Detailed content of the document

[1887] (Application example 1)

[1888] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1889] Conventional search systems often have difficulty quickly and accurately finding the information and products users are looking for. In particular, virtual stores often contain a large amount of product information, making it difficult for users to easily find the products they are looking for. Furthermore, there is a lack of product suggestion systems that effectively utilize natural language processing technology, which means they are unable to accurately understand users' search intent and suggest appropriate products.

[1890] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1891] In this invention, the server includes means for inputting materials, means for generating a feature vector from the materials, means for storing the generated feature vector in a database, means for receiving a search request from a user, means for generating a query feature vector from the search request, means for comparing the feature vector in the database with the feature vector of the query, means for selecting highly relevant materials based on the comparison results, means for displaying the selected materials, means for receiving a product search request from a user using a smart device and analyzing the search request, means for quantifying the query feature vector using a generative AI model, and means for comparing the query feature vector with the feature vector in the product database by similarity calculation and suggesting highly relevant products. This enables the user to quickly and accurately search for and suggest products they are looking for in a virtual store, significantly improving the user's shopping experience.

[1892] "Materials" means information or data that is collected, stored or managed for a specific purpose.

[1893] A "feature vector" is a numerical representation of the content of a document or query, making it easier to analyze and compare.

[1894] A "database" is a structured collection of information and a system for storing, managing, and retrieving data.

[1895] "User" means any person or entity that utilizes the System to enter search requests and obtain results.

[1896] A "generative AI model" is a model that uses artificial intelligence technology to generate feature vectors from materials or queries.

[1897] A "smart device" is an electronic device that can connect to the Internet and has multiple functions (e.g., a smartphone or a head-mounted display).

[1898] "Natural language processing technology" is a technology that uses computers to understand and process human language (natural language).

[1899] A "search request" is an instruction or query for searching information that a user inputs to a system.

[1900] A "query" is a keyword or phrase that a user enters into a system to search for a specific piece of information.

[1901] "Similarity calculation" is a method for calculating the similarity between the feature vector of a query and the feature vectors in a database.

[1902] "Products" are objects for sale that are searched and suggested within the virtual store.

[1903] A "suggestion system" is a system that selects and displays appropriate materials and products based on a user's search request.

[1904] "Relevance" is an indicator of how closely a query matches materials or products in a database.

[1905] As an embodiment of the present invention, a system is constructed as follows. The system is composed of the main elements of a server, a terminal, and a user. The server is responsible for managing materials and products and for search functions, and the terminal provides an interface for user input and display of results. Users use the terminal to search for materials and products.

[1906] System configuration

[1907] server

[1908] The server has the following features:

[1909] 1. Input of materials and products

[1910] The server receives text data of materials and products (e.g., product names, descriptions, attributes, prices, etc.) as input.

[1911] 2. Generate feature vectors

[1912] The server generates feature vectors from documents and products using a generative AI model (e.g., a model using TensorFlow or PyTorch).

[1913] 3. Storage in the database

[1914] The server stores the generated feature vectors and the materials and products themselves in a database, along with related metadata (such as registration date and time, category, etc.).

[1915] 4. Processing Search Requests

[1916] The server receives a search request from a user, analyzes the query using natural language processing techniques (e.g., spaCy or NLTK), and generates a corresponding feature vector.

[1917] 5. Similarity calculation

[1918] The server compares the feature vectors of the generated query with the feature vectors in the database using similarity calculations (for example, cosine similarity calculations using Scikit-learn) and selects materials and products with high matching scores.

[1919] 6. Submitting the results

[1920] The server transmits a list of the selected materials and products to the user's terminal and displays it visually.

[1921] Terminal

[1922] The terminal has the following features:

[1923] 1. Accepting input from the user

[1924] The terminal receives the user's search query (for example, "blue summer dress") from an input form and sends it to the server.

[1925] 2. Displaying the results

[1926] The terminal visually displays the search results received from the server to the user, including the product title, summary, price, and image.

[1927] User

[1928] A user uses the system in the following steps:

[1929] 1. Enter your search query

[1930] A user inputs a search query using a smart device (e.g., a smartphone or a head-mounted display).

[1931] 2. Check the search results

[1932] The user checks the search results displayed on the terminal, selects the required materials or products, and checks the details.

[1933] Specific examples

[1934] For example, when a user uses a smartphone to enter a search query such as "I want a blue summer dress," the device sends this query to a server. The server analyzes the query and generates a feature vector using a generative AI model. It compares this with the feature vectors of products stored in a database and displays the most suitable products (for example, "A lightweight, breathable blue summer dress") in a list ordered by relevance to the user. The user can then select a product from the results and view detailed information.

[1935] Prompt Sentence Examples

[1936] An example prompt for a generative AI model is:

[1937] Generate a feature vector based on the following product information: product name 'Blue summer dress', description 'Lightweight and breathable', attributes 'Color: Blue Season: Summer', price '3000 yen'.

[1938] In this way, users can quickly and accurately search for the materials and products they need and efficiently obtain the information they desire.

[1939] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1940] Step 1:

[1941] The server receives text data of materials and products as input. A user or administrator sends information about the materials and products (e.g., product name, description, attributes, price, etc.) to the server via an input device. This input data is in text format and includes a detailed description of the materials and products. The server prepares the received data to be passed to the generative AI model.

[1942] Step 2:

[1943] The server generates a feature vector from the received text data of the documents or products. Using a generative AI model (for example, a model using TensorFlow or PyTorch), the text data is converted into a feature vector. This feature vector is a numerical representation of the content of the documents or products, and is used for searching and comparison.

[1944] Step 3:

[1945] The server stores the generated feature vectors and the data of the materials and products themselves in a database. Related metadata (registration date and time, category, etc.) is also saved. This allows for quick retrieval of data in response to future search requests.

[1946] Step 4:

[1947] A user inputs a search query using a smart device (e.g., a smartphone or a head-mounted display). For example, the user inputs a query such as "I want a blue summer dress" into an input form. The device then sends this query to the server.

[1948] Step 5:

[1949] The server receives a search query from a user and analyzes it using natural language processing techniques (e.g., spaCy or NLTK). It extracts important keywords and phrases from the query and generates a corresponding feature vector, which helps understand the user's intent.

[1950] Step 6:

[1951] The server compares the feature vector of the generated query with the feature vectors in the database. It calculates the similarity between the query and the feature vectors of the materials or products using methods such as cosine similarity calculations using Scikit-learn. Based on this similarity, it selects materials or products with the highest matching scores.

[1952] Step 7:

[1953] The server generates a list of selected materials and products, sorts them by relevance, and sends the list to the user's device. The list includes product titles, summaries, prices, images, etc.

[1954] Step 8:

[1955] The terminal visually displays the list of materials and products received from the server to the user. The user can view the results and check detailed information as needed. Specifically, the user uses an interface that displays the product name, description, price, image, etc.

[1956] Step 9:

[1957] The user can select the desired product from the displayed list and check the detailed information. For example, they can view the detailed information of a "lightweight and breathable blue summer dress" and consider purchasing it.

[1958] Through the above processing steps, the user can quickly and accurately search for the desired materials or products and efficiently obtain the desired information.

[1959] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1960] As an embodiment of this invention, a system that combines document management and user emotion recognition is constructed. The system is composed of a server, a terminal, a user, and an emotion engine. The server is responsible for document management and search functions, while the terminal provides an interface for user input and result display. The user uses the terminal to search for documents, and the emotion engine is responsible for recognizing the user's emotions.

[1961] Program processing

[1962] Study and storage of materials

[1963] The server trains the generative AI model on new materials and stores them in storage.

[1964] 1. Inputting materials

[1965] The server receives text data of materials as input, such as PDF files or presentation files about marketing strategies.

[1966] 2. Generate feature vectors

[1967] The server uses a generative AI model to generate a feature vector from the document, which is a numerical representation of the document's content.

[1968] 3. Storage in the database

[1969] The server stores the generated feature vectors and the documents themselves in a database, along with the document metadata (creation date, creator, etc.).

[1970] Receiving requests from users

[1971] The user uses a dedicated terminal to input the image and keywords of the material to be searched.

[1972] 1. Enter your request

[1973] The user inputs a search query (keywords, sentences, etc.) into the input form of the search terminal. For example, the user inputs "materials related to marketing strategies for 2023."

[1974] 2. Sending the request

[1975] The terminal sends the entered query to the server, which sends the query as text data.

[1976] Emotion recognition

[1977] To recognize the user's emotions, the system is equipped with an emotion engine.

[1978] 1. Acquiring Emotion Data

[1979] When a user enters a query, the device captures the user's facial expressions and voice in real time through a camera, microphone, etc.

[1980] 2. Emotion Analysis

[1981] The server uses an emotion engine to analyze the captured data and recognize the user's emotions (e.g., joy, sadness, surprise, impatience, etc.).

[1982] Search for documents

[1983] The server analyzes the user's request and searches the database for the most suitable material.

[1984] 1. Parsing the query

[1985] The server analyzes the received query using natural language processing technology and understands the meaning of the query.

[1986] 2. Generate a feature vector for the query

[1987] The server generates a feature vector from the query, which is a numerical representation of the query content.

[1988] 3. Comparison of materials

[1989] The server compares the feature vectors of the documents stored in the database with the feature vectors of the query, for example, by calculating the cosine similarity to evaluate the similarity.

[1990] 4. Adjusting the fit

[1991] The server adjusts the selection criteria for materials based on the user's emotions recognized by the emotion engine. For example, if the user is in a hurry, the system will prioritize more intuitive and concise materials.

[1992] 5. Selection of the most suitable materials

[1993] The server selects materials with high similarity and creates an optimal list of materials that reflects the results of the emotion engine.

[1994] Material suggestions

[1995] The server transmits a list of the selected materials to the user's terminal.

[1996] 1. Creating a list of materials

[1997] The server sorts the documents in descending order of relevance and creates a list of them, including the title, summary, and creation date of each document.

[1998] 2. Submission of materials

[1999] The server sends the created list of materials to the user's terminal as text data.

[2000] 3. View the list

[2001] The terminal visually displays a list of received materials to the user, allowing the user to check the title, summary, creation date, etc. at a glance.

[2002] 4. Check the details

[2003] The user selects a document in the list and performs an action to check the details, for example, clicking on a specific document to go to its details page.

[2004] Specific examples

[2005] For example, when a company provides a document called "Marketing Strategy 2023" as input to the server, the following steps are executed:

[2006] Steps 1-3:

[2007] The server trains a generative AI model on this material, generates a feature vector for the material, and stores it in a database.

[2008] Step 4-5:

[2009] The user enters the query "materials related to marketing strategy for 2023" into the terminal, which then sends it to the server.

[2010] Steps 6-7:

[2011] The server analyzes the query, generates a feature vector for the query, and compares it with the material in the database.

[2012] Step 8:

[2013] The device captures the user's facial expressions and voice through a camera and microphone, and the server analyzes the user's emotions using an emotion engine. For example, if the user is feeling anxious, the system will adjust the materials it selects based on this information.

[2014] Steps 9-13:

[2015] The server selects highly relevant materials, including "Marketing Strategy 2023," generates a list of materials, and sends it to the user's device. The device displays the list, and the user can select a material from the list to check its details.

[2016] This system allows users to efficiently search and obtain the materials they need, and also allows them to receive optimal suggestions based on their emotions.

[2017] The processing flow will be explained below.

[2018] Program processing

[2019] Study and storage of materials

[2020] Step 1:

[2021] The server receives as input the text data of the document, for example a PDF file or a presentation file about a new marketing strategy.

[2022] Step 2:

[2023] The server passes the received documents to the generative AI model, which generates a feature vector from the documents. This feature vector is a numerical representation of the contents of the documents.

[2024] Step 3:

[2025] The server stores the generated feature vectors in a database, as well as the materials themselves and their metadata (creation date, creator, etc.).

[2026] Receiving requests from users

[2027] Step 4:

[2028] Users use a dedicated terminal to input images and keywords for the materials they are searching for, for example, "materials related to marketing strategies for 2023."

[2029] Step 5:

[2030] The terminal sends the entered query to the server, which sends the query as text data.

[2031] Emotion recognition

[2032] Step 6:

[2033] The device captures the user's facial expressions and voice data in real time through a camera and microphone while the user enters a query.

[2034] Step 7:

[2035] The server uses an emotion engine to analyze the captured data and recognize the user's emotions, for example, whether the user is anxious or calm.

[2036] Search for documents

[2037] Step 8:

[2038] The server analyzes the received query using natural language processing technology to understand its meaning, clarifying the meaning of keywords and phrases.

[2039] Step 9:

[2040] The server generates a feature vector from the query, which is a numerical representation of the query content.

[2041] Step 10:

[2042] The server compares the feature vectors in the database with the feature vectors of the query, for example by calculating cosine similarity to assess relevance.

[2043] Step 11:

[2044] The server adjusts the selection criteria for materials based on the user's emotions recognized by the emotion engine. For example, if the user is in a hurry, the server adjusts the selection criteria to prioritize intuitive and concise materials.

[2045] Step 12:

[2046] The server selects highly relevant materials and lists the most suitable materials based on the similarity between the query and the feature vectors of the materials.

[2047] Material suggestions

[2048] Step 13:

[2049] The server creates a list of the selected materials, sorted by relevance, including title, summary, and creation date.

[2050] Step 14:

[2051] The server sends the created list of materials to the user's terminal as text data.

[2052] Step 15:

[2053] The terminal visually displays the list of received materials to the user, including the title, summary, and creation date.

[2054] Step 16:

[2055] The user can select an item in the list to view its details, for example, by clicking on a specific item to go to its details page.

[2056] Specific examples

[2057] For example, when a company provides a document called "Marketing Strategy 2023" as input to the server, the following steps are executed:

[2058] Step 1:

[2059] The server receives text data of a document called "Marketing Strategy 2023."

[2060] Step 2:

[2061] The server passes the material to a generative AI model to generate a feature vector.

[2062] Step 3:

[2063] The server stores the generated feature vectors, the material itself, and its metadata in a database.

[2064] Step 4:

[2065] The user enters the query "Marketing strategy materials for 2023" into the terminal.

[2066] Step 5:

[2067] The terminal sends this query to the server.

[2068] Step 6:

[2069] The device uses a camera and microphone to capture the user's emotional data.

[2070] Step 7:

[2071] The server analyzes the user's emotions using an emotion engine and recognizes that the user is, for example, "anxious."

[2072] Steps 8-12:

[2073] The server analyzes the query, generates a feature vector, compares it with the database, and adjusts the selection criteria based on the user's sentiment to select the most suitable list of materials.

[2074] Steps 13-16:

[2075] The server generates a list of materials and sends it to the user's terminal, which displays it. The user selects a material and checks its details.

[2076] This system allows users to efficiently search and obtain the materials they need, and also allows them to receive optimal suggestions based on their emotions.

[2077] Example 2

[2078] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2079] Existing document management systems search and suggest documents without considering the user's emotions, making it difficult to intuitively provide the documents the user needs. In particular, the optimal documents may differ depending on the user's emotional state, which can lead to a poor user experience.

[2080] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for inputting materials, means for generating a feature vector from the materials, means for storing the generated feature vector in a database, means for receiving a search request from a user, means for generating a query feature vector from the search request, means for comparing the feature vector in the database with the feature vector of the query, means for selecting highly suitable materials based on the comparison results, means for displaying the selected materials, means for recognizing the user's emotions, and means for adjusting the material selection criteria based on the emotions. This makes it possible to search for and suggest optimal materials according to the user's emotional state.

[2081] "Materials" refers to documents or data in which information is organized in the form of text, images, audio, etc.

[2082] A "feature vector" is vector data that numerically represents the content of a document or the meaning of a query.

[2083] A "database" is a system that systematically stores and manages materials, feature vectors, metadata, etc.

[2084] A "user search request" refers to keywords or sentences that a user inputs to the system to search for specific materials.

[2085] A "query feature vector" is numerical vector data generated based on a search request.

[2086] The "comparison result" refers to the evaluation result of similarity or compatibility obtained by comparing the feature vectors in the database with the feature vectors of the query.

[2087] "Material selection criteria" refers to the criteria or indicators used to select materials, including sentiment and relevance.

[2088] "User's emotions" refers to the user's psychological or emotional state (for example, joy, sadness, surprise, impatience, etc.).

[2089] "Camera and microphone" refers to input devices for capturing the user's facial expressions and voice.

[2090] A "generative AI model" is a generative model that uses artificial intelligence and is an algorithm used to generate feature vectors from materials or queries.

[2091] MODE FOR CARRYING OUT THE INVENTION

[2092] As an embodiment of this invention, we will build a system that combines document management and user emotion recognition. The system is composed of a server, a terminal, a user, and an emotion engine. The server is responsible for document management and search functions, while the terminal provides an interface for user input and display of results. The user uses the terminal to search for documents, and the emotion engine is responsible for recognizing the user's emotions. Specific embodiments of each element of this system will be described below.

[2093] Study and storage of materials

[2094] The server trains new documents using a generative AI model. Specifically, the server receives the document's text data as input and generates a feature vector using a generative AI model (e.g., GPT-4). This feature vector is a numerical representation of the document's content and is stored in a database. The stored data includes not only the feature vector, but also the document itself and its metadata (creation date and time, creator, etc.).

[2095] Receiving requests from users

[2096] A user attempts to search for materials using a dedicated terminal. The user enters a search query (e.g., "materials related to marketing strategies for 2023") into the terminal's input form, and the terminal sends this query as text data to the server. The query is sent to the server as an HTTP request and processed by the server.

[2097] Emotion recognition

[2098] When a user enters a query, the device captures the user's facial expressions and voice in real time through a camera and microphone. This data is sent to a server, which then uses an emotion engine to analyze the user's emotions. For example, it uses Microsoft's Azure Face API or Google Cloud Natural Language emotion analysis tools to recognize the user's emotions (happiness, sadness, surprise, impatience, etc.).

[2099] Search for documents

[2100] The server analyzes the user's request and searches the database for the most appropriate materials. First, it analyzes the received query using natural language processing technology (e.g., BERT or GPT-4) to understand the meaning of the query. Then, it uses a generative AI model to generate a feature vector from the query. The server compares this feature vector with feature vectors in the database and evaluates the similarity, for example, by calculating cosine similarity. Furthermore, it adjusts the material selection criteria based on the user's emotions recognized by the emotion engine, and selects materials with high similarity.

[2101] Material suggestions

[2102] The server sends a list of selected materials to the user's device. The list is sorted in order of relevance and includes information such as the material's title, summary, and creation date. The device visually displays the received material list, and the user can select a material from the list to check its details. Specifically, clicking on the list on the device takes the user to the details page.

[2103] Specific examples

[2104] For example, if a company provides a document titled "Marketing Strategy 2023" as input, the server trains the document through a generative AI model, generates a feature vector, and stores it in a database. The user then enters a query, "Documents related to marketing strategy for 2023," and the device sends this to the server. The server analyzes the query, generates a specific feature vector, and compares it with documents in the database. The device captures the user's facial expressions and voice and analyzes their emotions using an emotion engine. For example, if the user is anxious, the system adjusts the documents it selects based on this information. Finally, the server selects documents with high relevance, including "Marketing Strategy 2023," generates a document list, and sends it to the device. The user can select a document from the list to view its details.

[2105] This system allows users to efficiently search and obtain the materials they need, and also allows them to receive optimal suggestions based on their emotions.

[2106] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2107] Step 1:

[2108] Inputting materials

[2109] The server receives text data of documents as input, such as PDF files or presentation files, which may be obtained from an internal management system or an external data source.

[2110] Input: Text data of the material (e.g., PDF file, Word file)

[2111] What it does: The server reads the document and extracts its contents as a string.

[2112] Output: Text data

[2113] Step 2:

[2114] Feature Vector Generation

[2115] The server uses a generative AI model (e.g., GPT-4) to generate a feature vector from the document. The feature vector is a numerical representation of the content of the document.

[2116] Input: Extracted text data

[2117] How it works: Text data is fed into a generative AI model to generate a feature vector.

[2118] Output: Feature vector

[2119] Step 3:

[2120] Storage in the database

[2121] The server stores the generated feature vectors and the materials themselves in a database together with metadata.

[2122] Input: feature vector, text data of the material, metadata (creation date, creator, etc.)

[2123] Operation: Issues an INSERT statement to the database and saves the data.

[2124] Output: Data is stored in the database

[2125] Step 4:

[2126] Entering a Request

[2127] The user enters a search query (for example, "materials related to marketing strategies for 2023") into the input form on the dedicated terminal.

[2128] Input: User's search query (keywords, sentences)

[2129] Action: Enter text using the keyboard on the device and click the send button.

[2130] Output: Text data of the search query

[2131] Step 5:

[2132] Submitting a request

[2133] The device sends the entered query to the server, sending data using an HTTP request.

[2134] Input: Text data of the search query

[2135] What it does: Sends a query to the server via an HTTP POST request.

[2136] Output: Query data sent to the server

[2137] Step 6:

[2138] Acquiring emotion data

[2139] When a user enters a query, the device captures the user's facial expressions and voice in real time through the camera and microphone.

[2140] Input: User's facial expression data, voice data

[2141] What it does: Captures a picture of your face with a webcam and records audio with a microphone.

[2142] Output: Captured facial expression data, audio data

[2143] Step 7:

[2144] Emotion Analysis

[2145] The server uses an emotion engine (e.g., Azure Face API) to analyze the captured data and recognize the user's emotions.

[2146] Input: Captured facial expression data, voice data

[2147] Operation: Facial images and voice data are input into the emotion engine to obtain emotional information.

[2148] Output: Emotion information (happiness, sadness, surprise, impatience, etc.)

[2149] Step 8:

[2150] Parsing a query

[2151] The server analyzes the received query using natural language processing technology (e.g., BERT or GPT-4) to understand the meaning of the query.

[2152] Input: Text data of the search query

[2153] How it works: Inputs query text into a natural language processing engine to perform analysis.

[2154] Output: Semantic information of the parsed query

[2155] Step 9:

[2156] Generate a query feature vector

[2157] The server generates a feature vector from the query.

[2158] Input: Parsed query semantics

[2159] How it works: Query information is input into the generative AI model, which generates a feature vector.

[2160] Output: Feature vector of the query

[2161] Step 10:

[2162] Comparison of materials

[2163] The server compares the feature vectors of the documents stored in the database with the feature vectors of the query, for example, by calculating the cosine similarity to evaluate the similarity.

[2164] Input: Query feature vector, database feature vector

[2165] Operation: Performs a cosine similarity calculation on each feature vector.

[2166] Output: Similarity score with each document

[2167] Step 11:

[2168] Adjusting the fit

[2169] The server adjusts the selection criteria for materials based on the user's emotions recognized by the emotion engine. If the user is in a hurry, intuitive and concise materials will be suggested first.

[2170] Input: Similarity score, emotion information

[2171] How it works: Adjust filtering conditions and score weighting based on emotional information.

[2172] Output: Adjusted similarity score

[2173] Step 12:

[2174] Selection of the most suitable materials

[2175] The server selects materials with high similarity and creates an optimal list of materials that reflects the results of the emotion engine.

[2176] Input: Adjusted similarity score

[2177] Operation: Select high-scoring materials and create a list.

[2178] Output: List of best materials

[2179] Step 13:

[2180] Creating a list of materials

[2181] The server sorts the documents in descending order of relevance and creates a list, including the document title, summary, creation date, etc.

[2182] Input: List of optimal materials

[2183] What it does: Sorts the list of documents and adds titles, summaries, creation dates, etc.

[2184] Output: Document list

[2185] Step 14:

[2186] Sending materials

[2187] The server sends the created document list to the user's terminal, and sends text data as an HTTP response.

[2188] Input: Document list

[2189] Operation: Returns the list of materials as an HTTP response in JSON format or similar.

[2190] Output: List of materials sent to the terminal

[2191] Step 15:

[2192] Viewing the list

[2193] The terminal visually displays a list of received materials to the user, allowing the user to check the title, summary, creation date, etc. at a glance.

[2194] Input: Document list

[2195] What it does: Renders a list visually with HTML and CSS.

[2196] Output: List of displayed materials

[2197] Step 16:

[2198] Check details

[2199] The user selects a document in the list and performs an action to check the details, for example, clicking on a specific document to go to its details page.

[2200] Input: Selection action on the material list

[2201] Action: Click on the document link to display the detailed view.

[2202] Output: Show detailed information

[2203] (Application example 2)

[2204] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2205] Conventional document management systems have the problem that when a user searches, the optimal document is not presented based on the user's emotions, making it difficult to effectively retrieve documents depending on the user's emotional state. Furthermore, search results may not fully satisfy the user's needs, resulting in a poor user experience. This is particularly important in virtual store environments, where users are required to quickly and accurately obtain products and information that match their emotions.

[2206] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting materials, means for generating a feature vector from the materials, means for storing the generated feature vector in a database, means for receiving a search request from a user, means for generating a query feature vector from the search request, means for comparing the feature vector in the database with the feature vector of the query, means for selecting materials with high suitability based on the comparison results, means for displaying the selected materials, means for recognizing the user's emotions, and means for adjusting the suitability based on the user's emotions. This makes it possible to quickly search for and suggest optimal materials and products that match the user's emotions, thereby improving the user experience.

[2207] "Materials" refers to information or data that users search or view.

[2208] A "feature vector" is vector-format data that quantifies the content of a document or query and characterizes that content.

[2209] A "database" is a collection of stored data that stores feature vectors and information for later retrieval and comparison.

[2210] A "search request" is a query or keyword entered by a user to search for a particular resource.

[2211] A "query" is a character string or a sentence in a natural language that a user uses to make a search request.

[2212] "Emotion" refers to a psychological state that can be recognized from a user's facial expression or voice, and includes states such as joy, sadness, surprise, and impatience.

[2213] "Relevance" is an index that indicates the relevance or match of a document to a user's search request or emotional state.

[2214] A "server" is a central computer that manages materials, performs searches, performs sentiment analysis, etc., and operates the entire system in conjunction with terminals.

[2215] A "terminal" is a device that provides an interface for users to search for and view materials.

[2216] An "emotion engine" is software or hardware that analyzes a user's emotions and recognizes emotions from facial expressions, voice, gestures, etc.

[2217] As an embodiment of the present invention, a system that combines document management and user emotion recognition is constructed. This system is composed of the following elements: a server, a terminal, a user, and an emotion engine.

[2218] 1. Server

[2219] The server is responsible for document management and search functions. It performs a series of processes, including inputting new documents, generating feature vectors, storing them in the database, and analyzing search requests. Specifically, it receives the document's text data, generates feature vectors using a generative AI model, and stores these in the database. It also receives search queries from users, analyzes them using natural language processing technology, and generates a query feature vector. It then compares the query feature vector with the feature vectors in the database to select documents with a high degree of relevance. During this process, the relevance is adjusted based on the user's emotions recognized by the emotion engine. By adjusting the relevance, it selects documents that best fit the user's emotional state.

[2220] 2. Terminal

[2221] The terminal provides an interface for users to input materials and make search requests. Users use the terminal to search for materials and display search results. The terminal captures the user's facial expressions and voice in real time through a camera and microphone and sends them to the server. When a user searches for materials, they enter a search query into the terminal's input form, which is sent to the server. The terminal visually displays the search results sent from the server, and the user can select the appropriate material from the list.

[2222] 3. Emotion Engine

[2223] The emotion engine recognizes the user's emotions. Specifically, when a user uses a device to input a query, it analyzes data captured through the device's camera and microphone to recognize the user's emotions. This allows the system to understand the user's emotional state. For example, if the user is in a hurry, the system will prioritize suggesting concise materials that can be understood immediately.

[2224] Specific examples

[2225] For example, in a VR shopping application for a virtual store, a user might say, "I'm looking for new sneakers." In this case, the device captures the user's facial expressions and voice in real time and sends them to the server. The server uses an emotion engine to analyze the user's emotions, and if it determines that the user is feeling anxious, it adjusts the suitability based on that information. As a result, sneakers that are most suitable for the user (popular items, highly reviewed items, etc.) are displayed preferentially.

[2226] Example prompts for generative AI models

[2227] "We suggest the best products for users who are in a hurry to find new sneakers."

[2228] This system allows users to efficiently search for and obtain the products they need in a VR environment and receive optimal suggestions based on their emotions at the time. The required hardware includes a VR headset, camera, and microphone. A typical VR headset (e.g., Meta Quest 2 or HTC Vive) is used as the specific device. Emotion analysis uses libraries such as Hume AI and Affectiva as emotion engines. On the server side, natural language processing technology (e.g., SpaCy and BERT) is used for document management and search.

[2229] In this way, the system of the invention can quickly search for and suggest the most suitable materials and products according to the user's emotional state, improving the user experience.

[2230] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2231] Step 1:

[2232] The user wears a VR headset and accesses a virtual store. The device captures the user's voice input, gaze, and facial expressions. As input, it acquires the user's voice query (e.g., "I'm looking for new sneakers") and emotional data including facial expressions and vocal tone.

[2233] Step 2:

[2234] The device extracts a search query from the voice input and sends it to the emotion engine. Specifically, it uses a microphone to convert the voice into text, which is then used as the query. The query is then sent to the server as text data.

[2235] Step 3:

[2236] The server analyzes the input query using natural language processing techniques (e.g., SpaCy, BERT) and generates a feature vector for the query. It receives the text data of the query as input and outputs the feature vector.

[2237] Step 4:

[2238] The device captures emotion data from the user's facial expressions and voice in real time and sends it to the emotion engine. Specifically, it uses a camera and microphone to capture the user's facial expressions and voice and sends the data to the server. The emotion data is output as extracted features.

[2239] Step 5:

[2240] The server analyzes the emotion data acquired by the emotion engine and recognizes the user's emotional state. It receives emotion data as input and outputs an emotional state such as joy, sadness, surprise, or impatience.

[2241] Step 6:

[2242] The server compares the query feature vector with the feature vectors in the database and calculates the similarity. Specifically, it uses mathematical techniques such as cosine similarity to compare the feature vectors of each document, and outputs the most suitable document from the database.

[2243] Step 7:

[2244] The server adjusts the relevance of search results based on the user's emotional state. For example, if the user is in a hurry, it will prioritize concise and intuitive materials. It receives the user's emotional state and the similarity of the materials as input, and outputs a list of materials with adjusted relevance.

[2245] Step 8:

[2246] The server selects the most relevant materials, creates a list of materials, and sends it to the user's terminal. The server receives the adjusted list of materials as input and sends it to the terminal as text data.

[2247] Step 9:

[2248] The device visually displays the received list of materials to the user. Specifically, the list is displayed on a display in the VR environment, and the user can select the appropriate material from the list. The output shows the title and summary of the material.

[2249] Step 10:

[2250] The user selects the document for which they wish to view details from the displayed list of documents. They are then taken to the details page of the selected document and can view the contents of the document. The system receives the user's selection as input and displays the detailed contents of the document as output.

[2251] Through this series of steps, users can efficiently search for and acquire the products they need in a VR environment and receive optimal suggestions based on their emotions at the time.

[2252] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2253] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data ...

Claims

1. A means for inputting data; means for generating a feature vector from the material; means for storing the generated feature vector in a database; means for receiving a search request from a user; means for generating a query feature vector from the search request; a means for comparing the feature vectors in the database with the query feature vector; A means for selecting highly relevant materials based on the comparison results; a means for displaying the selected materials; A system including:

2. The system of claim 1 , further comprising: a database for storing metadata about the material;

3. The system of claim 1 , wherein the system analyzes the query using natural language processing techniques.

Images