system

Abstract

We provide the system. [Solution] A system that includes a means of combining a generative AI and a lost and found management system, a means of responding to inquiries from the owner of a lost item in conversational form, and a means of enabling voice and image searches by combining voice AI and image AI.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a conventional lost-and-found management system, in order for the loser to identify their lost item, they need to have information about the specific item and input that information as text. However, if the loser does not accurately remember the information about the specific item or if text input is difficult, it has been difficult for them to identify their lost item.

Means for Solving the Problems

[0005] The system of this invention combines a generative AI with a lost and found management system to respond to inquiries from owners of lost items in conversational form. Furthermore, by combining it with voice AI and image AI, it enables searching by voice and image. This makes it possible for owners to identify their lost items even if they do not accurately remember the specific details of the item or if text input is difficult. [Brief explanation of the drawing]

[0006] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Embodiment 1 of Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1 of Form Example 1. [Figure 13]This is a sequence diagram showing the processing flow of the data processing system in Example 2 of Embodiment 2. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2 of Form Example 2. [Figure 15] This is a sequence diagram showing the processing flow of the data processing system in Embodiment 3 of Example 3. [Figure 16] This is a sequence diagram showing the processing flow of the data processing system in Application Example 3 of Form Example 3. [Figure 17] This is a sequence diagram showing the processing flow of the data processing system in Example 1 of the Form 1 when an emotion engine is combined. [Figure 18] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] This is a sequence diagram showing the processing flow of a data processing system in another embodiment. [Modes for carrying out the invention]

[0007] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0008] First, let's explain the terminology used in the following explanation.

[0009] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)), etc.

[0010] In the following embodiments, the numbered RAM (Random Access Memory) is a memory where information is temporarily stored and is used as a work memory by the processor.

[0011] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0012] In the following embodiments, the numbered communication I / F (Interface) is an interface that includes a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it may be only A, only B, or a combination of A and B. Also, in this specification, when expressing three or more matters connected by "and / or", the same concept as "A and / or B" is applied.

[0014] [First Embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] As shown in FIG. 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of the "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. Also, the database 24 and the communication I / F 26 are connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. Also, the reception device 38, the output device 40, and the camera 42 are connected to the bus 52.

[0019] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0022] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0026] Next, the identification process performed by the identification processing unit 290 of the data processing device 12 will be described.

[0027] "Example of form 1"

[0028] The system of the present invention combines a generative AI and a lost and found management system to respond to inquiries from owners of lost items in conversational form. Specifically, the generative AI analyzes the content of the inquiry from the owner, and based on that, the lost and found management system provides appropriate lost and found information. For example, if the owner inquires, "I forgot my red umbrella yesterday," the generative AI analyzes this inquiry, and the lost and found management system provides information about the "red umbrella" that was found "yesterday."

[0029] "Example of form 2"

[0030] Furthermore, the system of the present invention, when combined with voice AI and image AI, enables searching by voice and image. Specifically, when the owner of a lost item provides voice or image as input, the voice AI and image AI analyze it, and based on that, the lost and found management system provides appropriate lost and found information. For example, if the owner provides an image of an umbrella, the image AI analyzes this image, and the lost and found management system provides information about an umbrella that matches the image.

[0031] The following describes the processing flow for each example of the form.

[0032] "Example of form 1"

[0033] Step 1: The owner makes an inquiry in a conversational message. For example, they might say, "I left my red umbrella behind yesterday."

[0034] Step 2: The generative AI analyzes the content of this inquiry. Through this analysis, it extracts the information that the person forgot their "red umbrella" "yesterday".

[0035] Step 3: The lost and found management system provides appropriate lost and found information based on the analysis results. Specifically, it provides information about a "red umbrella" that was found "yesterday".

[0036] "Example of form 2"

[0037] Step 1: The owner provides audio or images as input. For example, the owner provides an image of an umbrella.

[0038] Step 2: Voice AI and image AI analyze the provided audio and images. This analysis extracts the features of the umbrella.

[0039] Step 3: The lost and found management system provides appropriate lost and found information based on the analysis results. Specifically, it provides information about an umbrella that matches the image.

[0040] (Example 1)

[0041] Next, we will describe Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0042] In managing lost and found items, it is essential to respond quickly and accurately to inquiries from owners. However, conventional systems have the problem of being inconvenient for users because they take a long time to analyze the content of inquiries and search for related information. In addition, there is a lack of search functions using voice and images, which prevents them from meeting the diverse needs of users.

[0043] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0044] In this invention, the server includes means for analyzing the content of an inquiry using an information processing device and natural language processing technology, means for searching for relevant information from a database based on the analysis results, and means for providing the search results to the user. This enables the user to quickly and accurately obtain information about lost items. Furthermore, by combining voice processing technology and image processing technology, searches using voice and images become possible, meeting the diverse needs of users.

[0045] An "information processing device" is a device used for inputting, processing, and outputting data, and includes computers and servers.

[0046] "Natural language processing technology" refers to technologies that enable computers to understand, interpret, and generate human language, and utilize text analysis and language models.

[0047] "Generative artificial intelligence" refers to artificial intelligence that has the ability to generate new information or content based on given data or prompts.

[0048] A "lost and found management device" is a device used to manage, search for, and provide information related to lost and found items.

[0049] "Speech processing technology" refers to technologies for analyzing, recognizing, converting, and generating audio data.

[0050] "Image processing technology" refers to the technology used to analyze, recognize, transform, and generate image data.

[0051] A "database" is a system for efficiently storing, searching, and managing data.

[0052] A "user" is an individual or organization that attempts to obtain information using the system.

[0053] This invention is a system that efficiently processes inquiries about lost items using an information processing device. The server utilizes generative artificial intelligence to analyze inquiries from users in natural language. Specifically, the server uses natural language processing technology to extract important keywords, dates, and item characteristics from the inquiry text. Generative AI models such as OpenAI's GPT-4 (registered trademark) can be used for this analysis.

[0054] Based on the analysis results, the server queries the lost and found management device and searches the database for relevant lost and found information. The database can efficiently retrieve information using SQL queries and other methods. The search results are provided to the user from the server via a terminal. The user can use the terminal to check detailed information about the lost and found item.

[0055] Furthermore, by combining voice processing and image processing technologies, it becomes possible to perform searches using voice and images. This allows users to make inquiries more intuitively through voice input and image uploads.

[0056] For example, if a user inquires, "I lost my blue wallet last Friday," the server uses a generative AI model to extract the keywords "last Friday" and "blue wallet." It then searches the lost and found system based on this information, and if a matching item is found, it provides the user with detailed information.

[0057] Examples of prompt messages include the following:

[0058] "User input: 'I lost my blue wallet last Friday.'"

[0059] "Generated AI prompt: 'Please search for information about the blue wallet found last Friday.'"

[0060] In this way, users can quickly and accurately obtain information about lost items.

[0061] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0062] Step 1:

[0063] The user uses their device to enter an inquiry about a lost item. The entered text data is sent from the user's device to the server. For example, if the user enters "I forgot my red umbrella yesterday," that text is sent to the server.

[0064] Step 2:

[0065] The server sends the received query to a generative AI model. The input is text data from the user. The server uses natural language processing technology to extract important keywords, dates, and item characteristics from the query. Specifically, the generative AI model analyzes keywords such as "yesterday" and "red umbrella," and obtains these keywords as output.

[0066] Step 3:

[0067] The server queries the lost and found management device based on the analysis results obtained from the generated AI model. The input is the analyzed keywords. The server uses an SQL query to search the database and identify information about a "red umbrella" found "yesterday". The output is detailed information about the corresponding lost item. Specifically, the server retrieves information about the corresponding lost item from the database.

[0068] Step 4:

[0069] The server provides users with information obtained from the lost and found management device. The input is information about lost items retrieved from a database. The server sends this information to the terminal, and the user can view detailed information about the relevant lost item through the terminal. Specifically, the information is displayed on the user's terminal.

[0070] (Application Example 1)

[0071] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server," and the smart device 14 will be referred to as a "terminal."

[0072] In modern brick-and-mortar stores, customers frequently forget items, but responding quickly and accurately to inquiries about lost items presents a challenge. In particular, there is a need to provide appropriate lost and found information in response to inquiries made in natural language.

[0073] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0074] In this invention, the server includes means for combining a generative information processing device and a lost and found management device, means for responding to inquiries from users in natural language, and means for enabling voice and image searches through a combination of a voice processing device and an image processing device. As a result, a user can make an inquiry about a lost item using a mobile information terminal, the generative information processing device will analyze the content of the inquiry, and the lost and found management device will quickly provide the relevant lost and found information.

[0075] A "generative information processing system" is an information processing system that analyzes queries in natural language and generates appropriate information.

[0076] A "lost and found item management device" is a device that manages information about lost and found items and provides the relevant lost and found item information as needed.

[0077] "User" refers to an individual or organization that uses the system to inquire about lost items.

[0078] "Natural language" refers to the language that humans use on a daily basis, and is the linguistic form that a system analyzes.

[0079] A "speech processing device" is a device used to analyze speech data and extract necessary information.

[0080] An "image processing device" is a device used to analyze image data and extract necessary information.

[0081] "Personal information terminals" refer to portable information processing devices such as smartphones and tablets.

[0082] "Inquiry content" refers to the content of questions or requests that users make to the system.

[0083] "Relevant lost and found information" refers to information about lost and found items identified based on the content of the inquiry.

[0084] To implement this invention, it is necessary to construct a system combining a server, a personal information terminal, a generative information processing device, a lost and found management device, a voice processing device, and an image processing device. The server uses the generative information processing device to analyze natural language inquiries from users and generate appropriate lost and found information. The personal information terminal functions as an interface for users to inquire about lost items.

[0085] Specifically, a user might use a mobile device to make an inquiry such as, "I forgot my blue wallet yesterday." The server sends this inquiry to a generative information processing device, which then analyzes the inquiry using a generative AI model. Based on the analysis results, the lost and found management device searches the database and identifies the relevant lost and found item. If voice and image processing devices are also included, searches using voice and image data are also possible.

[0086] This system allows users to quickly and accurately retrieve information about lost items. An example of a prompt message to the generating AI model is: "User inquiry: 'I forgot my blue wallet yesterday.' Extract the necessary information and generate keywords to query the lost and found management system."

[0087] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0088] Step 1:

[0089] A user uses a mobile device to inquire about a lost item. The input is natural language text, for example, "I forgot my blue wallet yesterday." The device sends this text to the server.

[0090] Step 2:

[0091] The server passes the received text to a generative information processing unit. A generative AI model analyzes this text and extracts important keywords (e.g., dates, object characteristics). The output is a list of the extracted keywords.

[0092] Step 3:

[0093] The server sends the extracted keywords to the lost and found device. The lost and found device searches the database and identifies the corresponding lost and found information. The input is a list of keywords, and the output is the corresponding lost and found information.

[0094] Step 4:

[0095] The server sends the information received from the lost and found device back to the mobile device. The device displays the relevant lost and found information to the user. The output is the lost and found information presented to the user.

[0096] Step 5:

[0097] If necessary, audio and image processing devices analyze the audio and image data to provide additional information. The input is audio or image data, and the output is the analyzed information.

[0098] (Example 2)

[0099] Next, we will describe Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0100] In managing lost and found items, there is a challenge in that it is difficult for owners to quickly and accurately identify their lost items using voice or images. Furthermore, conventional systems have limitations in voice and image-based searches, resulting in insufficient user convenience.

[0101] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0102] In this invention, the server includes means that combine generative artificial intelligence and a lost and found management device, means that enable searching using voice data and image data through a combination of voice processing artificial intelligence and image processing artificial intelligence, and means that search a database based on extracted feature information to identify lost and found information. This makes it possible for the owner of a lost item to quickly and accurately identify the lost item using voice and images.

[0103] "Generative artificial intelligence" is an artificial intelligence technology that generates appropriate responses and information based on user input.

[0104] A "lost and found management device" is a system for managing information about lost and found items and for searching and identifying them.

[0105] "Speech processing artificial intelligence" refers to artificial intelligence that possesses the technology to analyze speech data and convert it into text data.

[0106] "Image processing artificial intelligence" refers to artificial intelligence that possesses the technology to analyze image data and extract feature information.

[0107] "Audio data" refers to data that records audio in digital format.

[0108] "Image data" refers to data recorded in digital format.

[0109] "Feature information" refers to information extracted from images or audio to identify specific objects or content.

[0110] A "database" is a digital record system that systematically organizes information and allows for efficient searching and management.

[0111] A description of embodiments for carrying out this invention will be given.

[0112] Users utilize devices such as smartphones and personal computers to access the lost and found management system. Users can input voice or image descriptions of the lost item into their device. For example, a user might take a picture of an umbrella they lost at a train station and upload that image to the system.

[0113] The terminal transmits voice or image data input by the user to the server. The server uses speech processing artificial intelligence to convert the voice data into text data, and uses image processing artificial intelligence to analyze the image data and extract feature information. In this process, general speech recognition software is used for voice processing, and general image analysis software is used for image processing.

[0114] The server searches the database of the lost and found device based on the extracted characteristic information to identify the lost item. This allows users to obtain information about lost items quickly and accurately.

[0115] For example, if a user voice-inputs "I lost my blue wallet," the server uses voice processing artificial intelligence to extract the text "blue wallet" and sends it to the lost and found device. The system searches the database and returns information about the blue wallet to the server. The server provides this information to the user, who can then check how to retrieve their lost item at the lost and found center.

[0116] An example of a prompt message is, "Please analyze the image of the umbrella I lost at the station and provide information about the lost item." By using this prompt message, the user can give specific instructions to the system.

[0117] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0118] Step 1:

[0119] The user inputs audio or images describing the characteristics of the lost item into the device. For example, the user might take a picture of an umbrella they lost at a train station. This input data is saved on the device as audio or image data.

[0120] Step 2:

[0121] The terminal transmits voice or image data entered by the user to the server. A secure protocol is used over the internet for transmission, ensuring that the data reaches the server safely.

[0122] Step 3:

[0123] The server passes the received audio data to an AI for speech processing, which converts it into text data. The AI ​​uses speech recognition technology to analyze the audio data and generate the corresponding text. For example, the audio "I lost my blue wallet" is converted into the text "blue wallet".

[0124] Step 4:

[0125] The server passes the received image data to an image processing artificial intelligence (AI) to extract feature information. The AI ​​uses image analysis techniques to analyze the image data and identify the features of objects. For example, from an image of an umbrella, the feature information "black umbrella" is extracted.

[0126] Step 5:

[0127] The server searches the lost and found management device's database based on text data and feature information obtained from speech processing artificial intelligence and image processing artificial intelligence. The database search identifies the relevant lost and found information.

[0128] Step 6:

[0129] The server provides users with information about lost items retrieved from the database. Users can view detailed information about the lost items on their terminal screen and learn how to retrieve them. This allows users to quickly and accurately identify lost items.

[0130] (Application Example 2)

[0131] Next, we will describe Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server," and the smart device 14 will be referred to as a "terminal."

[0132] In physical stores, there is a challenge in quickly and accurately identifying and returning lost or forgotten items to their owners. Traditional methods often involve manual management of lost items, which is time-consuming and labor-intensive, and increases the risk of misidentification and overlooking information. This raises concerns about decreased customer satisfaction and negatively impacting the store's credibility.

[0133] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0134] In this invention, the server includes means that combine a generation system information processing device and a lost and found management device, means that enable searching by voice and image through a combination of a voice processing device and an image processing device, and means that input and analyze information on lost and found items using a portable information terminal or visual information device in a physical store. This enables the rapid and accurate identification and management of lost and found items in a physical store.

[0135] A "generative information processing device" is a device that executes algorithms and models for generating information, and in particular, a device that has the function of generating appropriate information based on user input.

[0136] A "lost and found management device" is a device that has the function of managing, identifying, and tracking information about lost and found items.

[0137] A "speech processing device" is a device that has the function of analyzing speech data and converting it into text data.

[0138] An "image processing device" is a device that has the function of analyzing image data and extracting features.

[0139] A "portable information terminal" is a portable information processing device used by users to input or retrieve information.

[0140] A "visual information device" is a device for acquiring and processing visual information, and in particular, a device that has the function of analyzing images and videos.

[0141] The invention will now be described in terms of embodiments for carrying out the invention. This invention is a system for streamlining the management of lost and found items in physical stores. The server comprises a generation system information processing device, a lost and found item management device, an audio processing device, and an image processing device. This makes it possible to identify and manage lost and found items using audio and images.

[0142] The user takes a picture of the found item or inputs a voice description using a mobile device or visual information device. The device sends this data to the server. The server converts the voice data to text using a voice processing device and analyzes the image data using an image processing device. Based on the analysis results, the found item management device searches the database for matching found item information and provides it to the user.

[0143] As a concrete example, a store staff member takes a picture of an umbrella they find in the store using a mobile device and uploads it to the system. The server uses an image processing device to analyze the umbrella's features, searches the database for matching lost item information, and provides it to the staff member. In this process, it is also possible to generate appropriate answers to user inquiries using a generative AI model.

[0144] An example of a prompt message would be: "Please provide lost and found information that matches this umbrella in the image."

[0145] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0146] Step 1:

[0147] The user takes a picture of the found item using a mobile device or visual information device and inputs a voice description. The input data consists of an image file and an audio file. The device sends this data to the server.

[0148] Step 2:

[0149] The server uses an audio processing unit to convert audio files into text data. The input is an audio file, and the output is text data. This process uses speech recognition technology to analyze the content of the audio and convert it into text information.

[0150] Step 3:

[0151] The server analyzes image files using an image processing device. The input is an image file, and the output is image feature data. This process uses image recognition technology to identify objects within the image and extract their features.

[0152] Step 4:

[0153] The server uses a lost and found management device to search the database for matching lost and found information based on the analyzed text data and image feature data. The input is text data and image feature data, and the output is lost and found information. This process uses database search technology to quickly identify information that matches the input data.

[0154] Step 5:

[0155] The server uses a generative AI model to generate appropriate answers to user inquiries. The input is the user's inquiry, and the output is the generated answer. This process uses natural language processing techniques to understand the user's intent and provide appropriate information.

[0156] Step 6:

[0157] The server sends the search results and generated answers to the terminal and provides them to the user. The input is information about the found item and the generated answers, and the output is the information provided to the user. This process uses communication technology to quickly deliver the information the user needs.

[0158] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0159] "Example of form 1"

[0160] One embodiment of the present invention includes an emotion engine in which a generative AI recognizes the emotions of the person who lost their item. Specifically, if the person who lost their item makes an inquiry saying, "I forgot my important umbrella yesterday," the emotion engine recognizes from the tone of voice and expression of the person that they are disappointed. It then conveys this information to the generative AI. Based on this information, the generative AI generates a response that will encourage the person who lost their item. For example, it might generate a response such as, "I'm sorry you forgot your important umbrella. Don't worry, we will do our best to find it for you."

[0161] "Example of form 2"

[0162] In another embodiment of the present invention, voice AI and image AI identify a lost item based on the owner's emotions. Specifically, the voice AI and image AI recognize the owner's anxiety from the tone of voice when the owner says "Find this" and from the accompanying image of the umbrella. Based on this information, the voice AI and image AI prioritize the search for the umbrella. For example, they search the database faster than usual and provide the owner with the results quickly.

[0163] The following describes the processing flow for each example of the form.

[0164] "Example of form 1"

[0165] Step 1: The owner of the lost item makes an inquiry saying, "I left my important umbrella behind yesterday."

[0166] Step 2: The emotional engine recognizes the owner's disappointment from the tone of their voice and expressions.

[0167] Step 3: The emotion engine transmits that information to the generative AI.

[0168] Step 4: The generative AI generates a response that will encourage the owner of the lost item, based on that information.

[0169] "Example of form 2"

[0170] Step 1: Provide a recording of the owner's voice saying "Please find this" along with an image of the umbrella provided.

[0171] Step 2: The voice AI and image AI recognize that the owner is panicking based on the tone of their voice and the image.

[0172] Step 3: The voice AI and image AI use that information to prioritize the search for umbrellas.

[0173] Step 4: Voice AI and image AI search the database faster than usual and quickly provide results to the owner.

[0174] (Example 1)

[0175] Next, we will describe Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0176] In inquiries regarding lost and found items, there is a need to improve user satisfaction by generating appropriate responses that take into account the user's feelings and enabling searches using voice and images. Conventional systems do not adequately consider the user's feelings when generating responses or by using voice and images for searches, which is a challenge.

[0177] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0178] In this invention, the server includes means for combining generative artificial intelligence and a lost and found management device, means for responding to inquiries from users in natural language, and means for analyzing the user's emotions using an emotion recognition device. This enables the generation of responses that take the user's emotions into consideration, and the retrieval of lost items using voice and images.

[0179] "Generative artificial intelligence" is an artificial intelligence technology that uses natural language processing techniques to analyze user inquiries and generate appropriate responses.

[0180] A "lost and found management device" is a device used to manage, search for, and provide information related to lost and found items.

[0181] "User" refers to an individual or organization that makes an inquiry regarding lost and found items.

[0182] "Natural language queries" refer to questions and requests made by users using everyday language.

[0183] An "emotion recognition device" is a device that analyzes and recognizes emotions from the tone and expression of a user's voice.

[0184] A "speech processing device" is a device used to analyze speech data and extract information.

[0185] An "image processing device" is a device used to analyze image data and extract information.

[0186] This invention uses a system that combines generative artificial intelligence and a lost and found management device to respond to user inquiries in natural language. The server utilizes generative artificial intelligence to analyze user inquiries. Specifically, it uses natural language processing technology to extract important information from the inquiry content. In this process, generative AI models such as OpenAI's GPT model can be used.

[0187] Based on the analysis results, the server uses a lost and found management device to search for information on the relevant lost item. The lost and found management device accesses the database and provides information about the lost item. Furthermore, the server uses an emotion recognition device to analyze the user's emotions. The emotion recognition device recognizes emotions from the tone and expression of the user's voice and transmits that information to the generative artificial intelligence.

[0188] Generative artificial intelligence generates encouraging responses to users based on recognized emotional information. For example, if a user inquires, "I forgot my important umbrella yesterday," the server will generate a response such as, "You forgot your important umbrella. Don't worry, we will do our best to find it for you."

[0189] For example, if a user inquires, "I forgot my red umbrella yesterday," the server will provide information about the "red umbrella" that was found "yesterday." Another example of a prompt message would be, "I forgot my red umbrella yesterday, has it been found?"

[0190] The flow of the specific processing in Example 1 will be explained using Figure 15.

[0191] Step 1:

[0192] A user uses a terminal to inquire about a lost item. The user inputs a question in natural language, such as "I left my red umbrella yesterday." The terminal then sends this inquiry to the server.

[0193] Step 2:

[0194] The server receives a query from the user. The server receives a natural language query sent from the terminal as input. The server uses a generative AI model to analyze the query. Specifically, it uses natural language processing techniques to extract important information such as "yesterday" and "red umbrella" from the query. The server then outputs the analyzed information.

[0195] Step 3:

[0196] The server utilizes the lost and found management system based on the analysis results. The input is information analyzed by a generating AI model. The server accesses the lost and found management system's database and searches for information on a "red umbrella" found "yesterday." The output is the corresponding lost and found information.

[0197] Step 4:

[0198] The server uses an emotion recognition device to analyze the user's emotions. The input includes the user's inquiry and tone of voice. Based on this information, the emotion recognition device analyzes the user's emotions and recognizes that they are disappointed. The output is the recognized emotion information.

[0199] Step 5:

[0200] The server uses a generative AI model to generate a response to encourage the user. It uses lost item information and emotional information as input. Based on this information, the generative AI model generates a response such as, "You've forgotten your important umbrella. Don't worry, we'll do our best to find it." The generated response is obtained as output.

[0201] Step 6:

[0202] The server sends the generated response to the user's terminal. The response generated by the generative AI model is used as input. The terminal displays this response to the user, allowing the user to verify it. The output is the response provided to the user.

[0203] (Application Example 1)

[0204] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server," and the smart device 14 will be referred to as a "terminal."

[0205] In modern brick-and-mortar stores, customers frequently leave items behind, requiring prompt and appropriate responses. However, traditional lost and found management systems struggle to address customer feelings, making improving customer satisfaction a challenge.

[0206] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0207] In this invention, the server includes means for combining a generative information processing device and a lost and found management device, means for responding to inquiries from users in natural language, and means for analyzing the user's emotions using an emotion recognition device. This makes it possible to provide lost and found information quickly and appropriately while taking into consideration the customer's emotions.

[0208] A "generative information processing device" is an information processing device that analyzes natural language queries from users and generates appropriate responses.

[0209] A "lost and found item management device" is a device that manages information related to lost and found items and provides that information as needed.

[0210] A "natural language query" refers to a question or request made by a user using the language they use on a daily basis.

[0211] A "speech processing device" is a device used to analyze speech data and extract necessary information.

[0212] An "image processing device" is a device used to analyze image data and extract necessary information.

[0213] An "emotion recognition device" is a device that analyzes a user's emotions from their voice and facial expressions and outputs the results.

[0214] "Means for generating responses" refers to means for creating appropriate responses for users based on analyzed information.

[0215] To implement this invention, a server needs to build a system that integrates a generative information processing device, a lost and found management device, a voice processing device, an image processing device, and an emotion recognition device. The server uses a generative AI model to analyze natural language queries from users and generate appropriate responses. Specifically, it uses OpenAI's GPT-3 (registered trademark) as the generative AI model and Microsoft Azure's (registered trademark) emotion analysis API for emotion recognition. The lost and found management device manages information about lost items using a database system such as MySQL (registered trademark).

[0216] The terminal functions as a smartphone or an in-store robot, receiving voice and text input from users. A voice processing unit converts voice data into text, and an image processing unit extracts necessary information from image data. This data is sent to a server and analyzed by a generative information processing unit.

[0217] For example, if a user asks the terminal, "I forgot my blue wallet yesterday," the server analyzes this information and searches the lost and found device for information on a "blue wallet" found "yesterday." If the emotion recognition device detects anxiety from the user's voice, the server generates a response such as, "Don't worry, we will do our best to find it."

[0218] An example of a prompt message would be: "A customer is inquiring about a blue wallet they left behind yesterday. Please generate a message to provide lost item information and reassure the customer."

[0219] The flow of a specific process in Application Example 1 will be explained using Figure 16.

[0220] Step 1:

[0221] The user makes a query to the device using natural language. The input is either the user's voice or text, which the device receives. In the case of voice, the voice processing unit converts the voice data into text. The output is the query in text format.

[0222] Step 2:

[0223] The terminal sends the query content in text format to the server. The server uses a generative AI model to analyze the query content. The input is the query content in text format, which the generative AI model analyzes and extracts relevant keywords and context. The output is the analyzed query content.

[0224] Step 3:

[0225] The server searches the lost and found device for the relevant lost and found information based on the analyzed query. The input is the analyzed query, which generates a database query and sends it to the lost and found device. The output is the relevant lost and found information.

[0226] Step 4:

[0227] The server uses an emotion recognition device to analyze the user's emotions. The input is emotion-related data extracted from the user's voice or text, which the emotion recognition device analyzes. The output is the user's emotional state.

[0228] Step 5:

[0229] The server uses a generative AI model to generate appropriate responses based on the information about the found item and the user's emotional state. The input consists of the information about the found item and the user's emotional state, which the generative AI model uses to generate the response. The output is the response message to the user.

[0230] Step 6:

[0231] The server sends the generated response message to the terminal. The terminal displays or audibly communicates this message to the user. The input is the response message, which the terminal communicates to the user. The output is the response information received by the user.

[0232] (Example 2)

[0233] Next, we will describe Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0234] While there is a need to efficiently and quickly identify lost items, conventional systems do not fully utilize voice and image-based searches, nor do they adjust priorities based on user sentiment. As a result, identifying lost items takes time, leading to decreased user satisfaction.

[0235] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0236] In this invention, the server includes means for combining an information processing device and a lost and found management device, means for analyzing voice data and image data from users, and means for enabling voice and image-based searching using a voice analysis device and an image analysis device. This makes it possible to efficiently identify lost and found items using voice and images.

[0237] An "information processing device" is a device used for inputting, analyzing, and outputting data, and has the function of processing audio data and image data.

[0238] A "lost and found management device" is a device that registers and manages information about lost and found items, and allows for searching and provision of that information as needed.

[0239] "User" refers to an individual or organization that inputs voice or images to search for lost and found items.

[0240] A "speech analysis device" is a device that has the function of analyzing speech data and converting it into text data.

[0241] An "image analysis device" is a device that analyzes image data and has the function of identifying objects and features within the image.

[0242] "Voice data" refers to information input by users via voice, and is the subject of analysis by a voice analysis device.

[0243] "Image data" refers to information input by users through images, and is the subject of analysis by image analysis devices.

[0244] "Search priority" is an indicator that shows the importance and urgency of searching for lost items, and is adjusted based on the user's feelings.

[0245] This invention relates to a lost and found management system using a voice analysis device and an image analysis device. The server combines an information processing device and a lost and found management device to analyze voice data and image data from users. Specifically, voice recognition software is used in the voice analysis device, and image recognition software is used in the image analysis device. This enables efficient identification of lost and found items using voice and images.

[0246] Users use devices such as smartphones or computers to input voice and image information to search for lost items. For example, a user might voice-input "I'm looking for a blue umbrella" and simultaneously upload an image of a blue umbrella to their device. The device then sends this data to the server.

[0247] The server uses a speech analysis device to convert speech data into text and analyze the user's request. It also uses an image analysis device to analyze image data and identify objects and features within the image. Based on this, the server searches the lost and found database and provides the user with appropriate lost and found information.

[0248] For example, if a user inputs "I'm looking for a blue umbrella" via voice input and provides an image of a blue umbrella, the server converts the voice into text using a voice analysis device and analyzes the image using an image analysis device. Based on these analysis results, the server searches its database and provides the user with information about blue umbrellas.

[0249] An example of a prompt message might be, "Upload an image of a blue umbrella and say 'I'm looking for a blue umbrella' aloud." In this way, users can efficiently search for lost items by combining voice and images.

[0250] The flow of the specific processing in Example 2 will be explained using Figure 17.

[0251] Step 1:

[0252] The user inputs voice and images into the device to search for the lost item. Specifically, the user speaks into the smartphone's microphone, "I'm looking for a blue umbrella," and simultaneously takes a picture of a blue umbrella with the camera and uploads it. The input consists of voice and image data, which form the basis for the next processing step.

[0253] Step 2:

[0254] The terminal transmits voice and image data entered by the user to the server. The data is encrypted and transmitted over the internet. The input is voice and image data from the terminal, and the output is the transmission of data to the server.

[0255] Step 3:

[0256] The server passes the received audio data to the speech analysis device. The speech analysis device uses speech recognition software to convert the audio to text and analyze the user's request. The input is audio data, and the output is text data. Specifically, the operation involves analyzing the audio waveform and converting it to text.

[0257] Step 4:

[0258] The server passes the received image data to the image analysis device. The image analysis device uses image recognition software to analyze the image and identify objects and features within it. The input is image data, and the output is feature information from the image. Specifically, it performs pattern recognition and feature extraction from the image.

[0259] Step 5:

[0260] The server searches the lost and found management system's database based on information obtained from the voice analysis device and the image analysis device. The input consists of text data and image feature information, and the output is information on matching lost and found items. Specifically, the server executes database queries and retrieves the results.

[0261] Step 6:

[0262] The server sends the search results to the terminal. The terminal displays the search results to the user. The input is the search result data from the server, and the output is the information displayed to the user. Specifically, the process involves formatting the results and displaying them on the screen.

[0263] (Application Example 2)

[0264] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0265] In modern brick-and-mortar stores, there is a need to quickly and accurately locate lost items and provide customers with the necessary information when they lose something on the premises. However, conventional lost and found management systems lack the ability to use voice or image-based searches, and they do not prioritize items based on customer sentiment, making it difficult to respond quickly.

[0266] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0267] In this invention, the server includes means for combining a generative AI and a lost and found management system, means for enabling voice and image searches through a combination of voice AI and image AI, and means for analyzing the user's emotions and adjusting the search priority. This makes it possible for customers to quickly search for lost items using voice and images and to be provided with information prioritized according to their emotions.

[0268] "Generative AI" is an artificial intelligence technology that generates appropriate responses and information based on user input.

[0269] A "Lost and Found Management System" is a system that manages information on lost and found items and provides that information to users as needed.

[0270] "Voice AI" is an artificial intelligence technology that analyzes voice data, converts it into text data, and reads emotions and intentions from voice.

[0271] "Image AI" is an artificial intelligence technology that analyzes image data to recognize objects and extract their features.

[0272] A "portable information device" is a portable information processing device such as a smartphone or tablet.

[0273] "Methods for analyzing user emotions and adjusting search priority" refers to technologies that analyze user emotions from voice and input data and dynamically change the priority of information retrieval based on the results.

[0274] The system for carrying out this invention consists of a personal information terminal and a server. The personal information terminal has an application installed that accepts voice input and image input. This application allows the user to input information about a lost item using voice or images.

[0275] The server uses speech AI and image AI to convert speech data received from users into text data using the Google® Cloud Speech-to-Text API, and analyzes image data using the Google Cloud Vision API. Furthermore, it uses IBM Watson® Tone Analyzer to analyze the emotions from the user's speech and adjust search priorities accordingly.

[0276] The analyzed data is processed by a generative AI and searches the database of the lost and found management system. This allows users to quickly obtain information about lost items and display the results on their mobile devices.

[0277] As a specific example, when a user loses an umbrella in a store, the user takes a picture of the umbrella with a portable information terminal and inputs "looking for an umbrella" by voice. At this time, as examples of prompt sentences, inputs such as "Please look for the umbrella in this image. I'm in a hurry." or "Please find the umbrella lost in the store." can be considered. The server analyzes these inputs and provides appropriate lost item information to the user.

[0278] The flow of the specific process in Application Example 2 will be described with reference to FIG. 18.

[0279] Step 1:

[0280] The user takes a picture of the lost item using a portable information terminal and inputs "looking for an umbrella" by voice. The input image data and voice data are transmitted to the server by the application.

[0281] Step 2:

[0282] The server converts the received voice data into text data using the Google Cloud Speech-to-Text API. By this process, the voice data is converted into a prompt sentence in text format.

[0283] Step 3:

[0284] The server analyzes the received image data using the Google Cloud Vision API. By this analysis, the objects and features in the image are identified and output as text data.

[0285] Step 4:

[0286] The server uses the IBM Watson Tone Analyzer to analyze the user's emotion from the voice data. Based on the analysis result, data for adjusting the search priority is generated.

[0287] Step 5:

[0288] The server uses generative AI to search the lost and found management system database based on text data and image analysis results. This search identifies information about the lost item.

[0289] Step 6:

[0290] The server sends the search results to the mobile device. The device displays information about the lost item to the user, enabling a quick response.

[0291] (Other examples)

[0292] Next, other embodiments will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0293] In modern society, users are expected to be quickly and accurately directed to the appropriate service when making inquiries. However, conventional systems are insufficient in analyzing inquiry content and considering emotional states, making it difficult to provide appropriate responses that meet user needs. Furthermore, identifying lost items using voice and image data is not efficient. This leads to challenges such as decreased user satisfaction and a decline in service quality.

[0294] The identification process performed by the identification processing unit 290 of the data processing device 12 in other embodiments is realized by the following means.

[0295] In this invention, the server includes means for receiving inquiries from users and generating prompt sentences using a generative AI model to analyze their content; means for analyzing audio and image data by combining audio processing and image processing technologies to identify lost items by extracting feature information; and means for using an emotion engine to analyze the user's emotions and generate responses based on their emotional state. This makes it possible to sort users into appropriate services based on their inquiries, efficiently identify lost items using audio and image data, and provide appropriate responses according to the user's emotional state.

[0296] A "generative AI model" is an artificial intelligence model that analyzes user inquiries and generates prompt messages to categorize them into the appropriate service.

[0297] A "prompt statement" is a sentence used to instruct a generative AI model to analyze the query and identify the appropriate service.

[0298] "Speech processing technology" is a technology for analyzing speech data and converting it into text.

[0299] "Image processing technology" refers to techniques for analyzing image data and extracting feature information.

[0300] An "emotion engine" is a system that analyzes a user's emotional state and generates a response based on the results.

[0301] "Feature information" refers to important information extracted from image data to identify found objects.

[0302] A description of the embodiment for carrying out the invention will be provided.

[0303] This invention is a system that efficiently processes user inquiries and categorizes them into appropriate services. The system operates between a server, a terminal, and a user.

[0304] The server receives inquiries from users. The inquiries are sent in text, voice, or image format. The server has set up an API endpoint via an HTTP request to properly receive this data.

[0305] To analyze the received inquiry content, the server uses a generative AI model. Specifically, a generative AI model such as OpenAI's GPT series is used to generate a prompt text for instructing the appropriate sorting of the inquiry content into services. An example of the prompt text is in the form of "Please identify the appropriate service based on the user's inquiry content."

[0306] The server inputs the generated prompt text into the generative AI model to analyze the inquiry content. The generative AI model executes instructions for sorting the inquiry content into appropriate services based on the prompt text. The analysis results are saved in the database within the server.

[0307] Furthermore, the server uses voice processing technology (e.g., Google Cloud Speech-to-Text) to convert voice data into text, and uses image processing technology (e.g., Amazon Rekognition) to analyze image data and extract feature information. As a result, text information is obtained from voice data, and feature information is obtained from image data.

[0308] The server searches its internal database based on the extracted feature information to identify lost items. The database stores past lost item information, and the lost items are identified by searching for data that matches the feature information.

[0309] Also, the server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the user's emotional state. Based on the analysis results, a response corresponding to the user's emotional state is generated. For example, when the user feels anxious, a response that reassures the user is generated.

[0310] The user's device receives a response from the server and displays it to the user. The user checks the response from the server displayed on the device and decides on their next action. For example, if the lost item is identified, the user may take action such as going to the designated location.

[0311] The flow of a specific process in another embodiment will be explained using Figure 19.

[0312] Step 1:

[0313] The server receives queries from users. Input is data in text, audio, or image format. The server receives this data via HTTP requests and temporarily stores the query content. Output is the storage of the query data.

[0314] Step 2:

[0315] The server uses a generative AI model to analyze the received text-based query. The input is text data received from the user. The server generates a prompt message for the generative AI model, instructing it to sort the query into the appropriate service. This prompt message is in the format of "Identify the appropriate service based on the user's query." The output is the generated prompt message.

[0316] Step 3:

[0317] The server inputs the generated prompt message into the generation AI model, which then analyzes the query. The input is the prompt message. The generation AI model analyzes the query based on the prompt message and executes instructions to sort it into the appropriate service. The output is the analysis result.

[0318] Step 4:

[0319] The server converts audio data into text using speech processing technology. The input is audio data received from the user. The server uses Google Cloud Speech-to-Text to analyze the audio data and extract text information. The output is the text information extracted from the audio data.

[0320] Step 5:

[0321] The server analyzes image data using image processing technology and extracts feature information. The input is image data received from the user. The server uses Amazon Rekognition to analyze the image data and extract feature information. The output is the feature information extracted from the image data.

[0322] Step 6:

[0323] The server searches its internal database based on the extracted feature information to identify the found item. The input is feature information. The server compares the feature information with past found item information in the database and searches for matching data. The output is the identified found item information.

[0324] Step 7:

[0325] The server analyzes the user's emotional state using an emotion engine. The input is the user's inquiry. The server uses IBM Watson Tone Analyzer to evaluate the emotional state and obtain the analysis results. The output is the analysis results regarding the user's emotional state.

[0326] Step 8:

[0327] The server generates a response based on the user's emotional state. The input is the result of the emotional state analysis. Based on the analysis, the server generates a response that will reassure the user. The output is the generated response.

[0328] Step 9:

[0329] The server sends the generated response to the user's terminal. The input is the generated response. The server sends the response via HTTP response so that it is displayed on the user's terminal. The output is the response displayed on the user's terminal.

[0330] Step 10:

[0331] The user checks the response from the server displayed on their terminal. The input is the response displayed on the terminal. The user decides on their next action based on the response. The output is the user's next action.

[0332] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0333] Data generation model 58 is a form of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0334] Other examples of generative AI include Gemini® (registered trademark) (Internet search). <url: https: gemini.google.com ?hl="ja">) are some examples.

[0335] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0336] [Second Embodiment]

[0337] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0338] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0339] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0340] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0341] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0342] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0343] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0344] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0345] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0346] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0347] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0348] Next, the identification process performed by the identification processing unit 290 of the data processing device 12 will be described.

[0349] "Example of form 1"

[0350] The system of the present invention combines a generative AI and a lost and found management system to respond to inquiries from owners of lost items in conversational form. Specifically, the generative AI analyzes the content of the inquiry from the owner, and based on that, the lost and found management system provides appropriate lost and found information. For example, if the owner inquires, "I forgot my red umbrella yesterday," the generative AI analyzes this inquiry, and the lost and found management system provides information about the "red umbrella" that was found "yesterday."

[0351] "Example of form 2"

[0352] Furthermore, the system of the present invention, when combined with voice AI and image AI, enables searching by voice and image. Specifically, when the owner of a lost item provides voice or image as input, the voice AI and image AI analyze it, and based on that, the lost and found management system provides appropriate lost and found information. For example, if the owner provides an image of an umbrella, the image AI analyzes this image, and the lost and found management system provides information about an umbrella that matches the image.

[0353] The following describes the processing flow for each example of the form.

[0354] "Example of form 1"

[0355] Step 1: The owner makes an inquiry in a conversational message. For example, they might say, "I left my red umbrella behind yesterday."

[0356] Step 2: The generative AI analyzes the content of this inquiry. Through this analysis, it extracts the information that the person forgot their "red umbrella" "yesterday".

[0357] Step 3: The lost and found management system provides appropriate lost and found information based on the analysis results. Specifically, it provides information about a "red umbrella" that was found "yesterday".

[0358] "Example of form 2"

[0359] Step 1: The owner provides audio or images as input. For example, the owner provides an image of an umbrella.

[0360] Step 2: Voice AI and image AI analyze the provided audio and images. This analysis extracts the features of the umbrella.

[0361] Step 3: The lost and found management system provides appropriate lost and found information based on the analysis results. Specifically, it provides information about an umbrella that matches the image.

[0362] (Example 1)

[0363] Next, we will describe Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0364] In managing lost and found items, it is essential to respond quickly and accurately to inquiries from owners. However, conventional systems have the problem of being inconvenient for users because they take a long time to analyze the content of inquiries and search for related information. In addition, there is a lack of search functions using voice and images, which prevents them from meeting the diverse needs of users.

[0365] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0366] In this invention, the server includes means for analyzing the content of an inquiry using an information processing device and natural language processing technology, means for searching for relevant information from a database based on the analysis results, and means for providing the search results to the user. This enables the user to quickly and accurately obtain information about lost items. Furthermore, by combining voice processing technology and image processing technology, searches using voice and images become possible, meeting the diverse needs of users.

[0367] An "information processing device" is a device used for inputting, processing, and outputting data, and includes computers and servers.

[0368] "Natural language processing technology" refers to technologies that enable computers to understand, interpret, and generate human language, and utilize text analysis and language models.

[0369] "Generative artificial intelligence" refers to artificial intelligence that has the ability to generate new information or content based on given data or prompts.

[0370] A "lost and found management device" is a device used to manage, search for, and provide information related to lost and found items.

[0371] "Speech processing technology" refers to technologies for analyzing, recognizing, converting, and generating audio data.

[0372] "Image processing technology" refers to the technology used to analyze, recognize, transform, and generate image data.

[0373] A "database" is a system for efficiently storing, searching, and managing data.

[0374] A "user" is an individual or organization that attempts to obtain information using the system.

[0375] This invention is a system that efficiently processes inquiries about lost items using an information processing device. The server utilizes generative artificial intelligence to analyze inquiries from users in natural language. Specifically, the server uses natural language processing technology to extract important keywords, dates, and item characteristics from the inquiry text. Generative AI models such as OpenAI's GPT-4 can be used for this analysis.

[0376] Based on the analysis results, the server queries the lost and found management device and searches the database for relevant lost and found information. The database can efficiently retrieve information using SQL queries and other methods. The search results are provided to the user from the server via a terminal. The user can use the terminal to check detailed information about the lost and found item.

[0377] Furthermore, by combining voice processing and image processing technologies, it becomes possible to perform searches using voice and images. This allows users to make inquiries more intuitively through voice input and image uploads.

[0378] For example, if a user inquires, "I lost my blue wallet last Friday," the server uses a generative AI model to extract the keywords "last Friday" and "blue wallet." It then searches the lost and found system based on this information, and if a matching item is found, it provides the user with detailed information.

[0379] Examples of prompt messages include the following:

[0380] "User input: 'I lost my blue wallet last Friday.'"

[0381] "Generated AI prompt: 'Please search for information about the blue wallet found last Friday.'"

[0382] In this way, users can quickly and accurately obtain information about lost items.

[0383] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0384] Step 1:

[0385] The user uses their device to enter an inquiry about a lost item. The entered text data is sent from the user's device to the server. For example, if the user enters "I forgot my red umbrella yesterday," that text is sent to the server.

[0386] Step 2:

[0387] The server sends the received query to a generative AI model. The input is text data from the user. The server uses natural language processing technology to extract important keywords, dates, and item characteristics from the query. Specifically, the generative AI model analyzes keywords such as "yesterday" and "red umbrella," and obtains these keywords as output.

[0388] Step 3:

[0389] The server queries the lost and found management device based on the analysis results obtained from the generated AI model. The input is the analyzed keywords. The server uses an SQL query to search the database and identify information about a "red umbrella" found "yesterday". The output is detailed information about the corresponding lost item. Specifically, the server retrieves information about the corresponding lost item from the database.

[0390] Step 4:

[0391] The server provides users with information obtained from the lost and found management device. The input is information about lost items retrieved from a database. The server sends this information to the terminal, and the user can view detailed information about the relevant lost item through the terminal. Specifically, the information is displayed on the user's terminal.

[0392] (Application Example 1)

[0393] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0394] In modern brick-and-mortar stores, customers frequently forget items, but responding quickly and accurately to inquiries about lost items presents a challenge. In particular, there is a need to provide appropriate lost and found information in response to inquiries made in natural language.

[0395] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0396] In this invention, the server includes means for combining a generative information processing device and a lost and found management device, means for responding to inquiries from users in natural language, and means for enabling voice and image searches through a combination of a voice processing device and an image processing device. As a result, a user can make an inquiry about a lost item using a mobile information terminal, the generative information processing device will analyze the content of the inquiry, and the lost and found management device will quickly provide the relevant lost and found information.

[0397] A "generative information processing system" is an information processing system that analyzes queries in natural language and generates appropriate information.

[0398] A "lost and found item management device" is a device that manages information about lost and found items and provides the relevant lost and found item information as needed.

[0399] "User" refers to an individual or organization that uses the system to inquire about lost items.

[0400] "Natural language" refers to the language that humans use on a daily basis, and is the linguistic form that a system analyzes.

[0401] A "speech processing device" is a device used to analyze speech data and extract necessary information.

[0402] An "image processing device" is a device used to analyze image data and extract necessary information.

[0403] "Personal information terminals" refer to portable information processing devices such as smartphones and tablets.

[0404] "Inquiry content" refers to the content of questions or requests that users make to the system.

[0405] "Relevant lost and found information" refers to information about lost and found items identified based on the content of the inquiry.

[0406] To implement this invention, it is necessary to construct a system combining a server, a personal information terminal, a generative information processing device, a lost and found management device, a voice processing device, and an image processing device. The server uses the generative information processing device to analyze natural language inquiries from users and generate appropriate lost and found information. The personal information terminal functions as an interface for users to inquire about lost items.

[0407] Specifically, a user might use a mobile device to make an inquiry such as, "I forgot my blue wallet yesterday." The server sends this inquiry to a generative information processing device, which then analyzes the inquiry using a generative AI model. Based on the analysis results, the lost and found management device searches the database and identifies the relevant lost and found item. If voice and image processing devices are also included, searches using voice and image data are also possible.

[0408] This system allows users to quickly and accurately retrieve information about lost items. An example of a prompt message to the generating AI model is: "User inquiry: 'I forgot my blue wallet yesterday.' Extract the necessary information and generate keywords to query the lost and found management system."

[0409] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0410] Step 1:

[0411] A user uses a mobile device to inquire about a lost item. The input is natural language text, for example, "I forgot my blue wallet yesterday." The device sends this text to the server.

[0412] Step 2:

[0413] The server passes the received text to a generative information processing unit. A generative AI model analyzes this text and extracts important keywords (e.g., dates, object characteristics). The output is a list of the extracted keywords.

[0414] Step 3:

[0415] The server sends the extracted keywords to the lost and found device. The lost and found device searches the database and identifies the corresponding lost and found information. The input is a list of keywords, and the output is the corresponding lost and found information.

[0416] Step 4:

[0417] The server sends the information received from the lost and found device back to the mobile device. The device displays the relevant lost and found information to the user. The output is the lost and found information presented to the user.

[0418] Step 5:

[0419] If necessary, audio and image processing devices analyze the audio and image data to provide additional information. The input is audio or image data, and the output is the analyzed information.

[0420] (Example 2)

[0421] Next, we will describe Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0422] In managing lost and found items, there is a challenge in that it is difficult for owners to quickly and accurately identify their lost items using voice or images. Furthermore, conventional systems have limitations in voice and image-based searches, resulting in insufficient user convenience.

[0423] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0424] In this invention, the server includes means that combine generative artificial intelligence and a lost and found management device, means that enable searching using voice data and image data through a combination of voice processing artificial intelligence and image processing artificial intelligence, and means that search a database based on extracted feature information to identify lost and found information. This makes it possible for the owner of a lost item to quickly and accurately identify the lost item using voice and images.

[0425] "Generative artificial intelligence" is an artificial intelligence technology that generates appropriate responses and information based on user input.

[0426] A "lost and found management device" is a system for managing information about lost and found items and for searching and identifying them.

[0427] "Speech processing artificial intelligence" refers to artificial intelligence that possesses the technology to analyze speech data and convert it into text data.

[0428] "Image processing artificial intelligence" refers to artificial intelligence that possesses the technology to analyze image data and extract feature information.

[0429] "Audio data" refers to data that records audio in digital format.

[0430] "Image data" refers to data recorded in digital format.

[0431] "Feature information" refers to information extracted from images or audio to identify specific objects or content.

[0432] A "database" is a digital record system that systematically organizes information and allows for efficient searching and management.

[0433] A description of embodiments for carrying out this invention will be given.

[0434] Users utilize devices such as smartphones and personal computers to access the lost and found management system. Users can input voice or image descriptions of the lost item into their device. For example, a user might take a picture of an umbrella they lost at a train station and upload that image to the system.

[0435] The terminal transmits voice or image data input by the user to the server. The server uses speech processing artificial intelligence to convert the voice data into text data, and uses image processing artificial intelligence to analyze the image data and extract feature information. In this process, general speech recognition software is used for voice processing, and general image analysis software is used for image processing.

[0436] The server searches the database of the lost and found device based on the extracted characteristic information to identify the lost item. This allows users to obtain information about lost items quickly and accurately.

[0437] For example, if a user voice-inputs "I lost my blue wallet," the server uses voice processing artificial intelligence to extract the text "blue wallet" and sends it to the lost and found device. The system searches the database and returns information about the blue wallet to the server. The server provides this information to the user, who can then check how to retrieve their lost item at the lost and found center.

[0438] An example of a prompt message is, "Please analyze the image of the umbrella I lost at the station and provide information about the lost item." By using this prompt message, the user can give specific instructions to the system.

[0439] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0440] Step 1:

[0441] The user inputs audio or images describing the characteristics of the lost item into the device. For example, the user might take a picture of an umbrella they lost at a train station. This input data is saved on the device as audio or image data.

[0442] Step 2:

[0443] The terminal transmits voice or image data entered by the user to the server. A secure protocol is used over the internet for transmission, ensuring that the data reaches the server safely.

[0444] Step 3:

[0445] The server passes the received audio data to an AI for speech processing, which converts it into text data. The AI ​​uses speech recognition technology to analyze the audio data and generate the corresponding text. For example, the audio "I lost my blue wallet" is converted into the text "blue wallet".

[0446] Step 4:

[0447] The server passes the received image data to an image processing artificial intelligence (AI) to extract feature information. The AI ​​uses image analysis techniques to analyze the image data and identify the features of objects. For example, from an image of an umbrella, the feature information "black umbrella" is extracted.

[0448] Step 5:

[0449] The server searches the lost and found management device's database based on text data and feature information obtained from speech processing artificial intelligence and image processing artificial intelligence. The database search identifies the relevant lost and found information.

[0450] Step 6:

[0451] The server provides users with information about lost items retrieved from the database. Users can view detailed information about the lost items on their terminal screen and learn how to retrieve them. This allows users to quickly and accurately identify lost items.

[0452] (Application Example 2)

[0453] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0454] In physical stores, there is a challenge in quickly and accurately identifying and returning lost or forgotten items to their owners. Traditional methods often involve manual management of lost items, which is time-consuming and labor-intensive, and increases the risk of misidentification and overlooking information. This raises concerns about decreased customer satisfaction and negatively impacting the store's credibility.

[0455] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0456] In this invention, the server includes means that combine a generation system information processing device and a lost and found management device, means that enable searching by voice and image through a combination of a voice processing device and an image processing device, and means that input and analyze information on lost and found items using a portable information terminal or visual information device in a physical store. This enables the rapid and accurate identification and management of lost and found items in a physical store.

[0457] A "generative information processing device" is a device that executes algorithms and models for generating information, and in particular, a device that has the function of generating appropriate information based on user input.

[0458] A "lost and found management device" is a device that has the function of managing, identifying, and tracking information about lost and found items.

[0459] A "speech processing device" is a device that has the function of analyzing speech data and converting it into text data.

[0460] An "image processing device" is a device that has the function of analyzing image data and extracting features.

[0461] A "portable information terminal" is a portable information processing device used by users to input or retrieve information.

[0462] A "visual information device" is a device for acquiring and processing visual information, and in particular, a device that has the function of analyzing images and videos.

[0463] The invention will now be described in terms of embodiments for carrying out the invention. This invention is a system for streamlining the management of lost and found items in physical stores. The server comprises a generation system information processing device, a lost and found item management device, an audio processing device, and an image processing device. This makes it possible to identify and manage lost and found items using audio and images.

[0464] The user takes a picture of the found item or inputs a voice description using a mobile device or visual information device. The device sends this data to the server. The server converts the voice data to text using a voice processing device and analyzes the image data using an image processing device. Based on the analysis results, the found item management device searches the database for matching found item information and provides it to the user.

[0465] As a concrete example, a store staff member takes a picture of an umbrella they find in the store using a mobile device and uploads it to the system. The server uses an image processing device to analyze the umbrella's features, searches the database for matching lost item information, and provides it to the staff member. In this process, it is also possible to generate appropriate answers to user inquiries using a generative AI model.

[0466] An example of a prompt message would be: "Please provide lost and found information that matches this umbrella in the image."

[0467] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0468] Step 1:

[0469] The user takes a picture of the found item using a mobile device or visual information device and inputs a voice description. The input data consists of an image file and an audio file. The device sends this data to the server.

[0470] Step 2:

[0471] The server uses an audio processing unit to convert audio files into text data. The input is an audio file, and the output is text data. This process uses speech recognition technology to analyze the content of the audio and convert it into text information.

[0472] Step 3:

[0473] The server analyzes image files using an image processing device. The input is an image file, and the output is image feature data. This process uses image recognition technology to identify objects within the image and extract their features.

[0474] Step 4:

[0475] The server uses a lost and found management device to search the database for matching lost and found information based on the analyzed text data and image feature data. The input is text data and image feature data, and the output is lost and found information. This process uses database search technology to quickly identify information that matches the input data.

[0476] Step 5:

[0477] The server uses a generative AI model to generate appropriate answers to user inquiries. The input is the user's inquiry, and the output is the generated answer. This process uses natural language processing techniques to understand the user's intent and provide appropriate information.

[0478] Step 6:

[0479] The server sends the search results and generated answers to the terminal and provides them to the user. The input is information about the found item and the generated answers, and the output is the information provided to the user. This process uses communication technology to quickly deliver the information the user needs.

[0480] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0481] "Example of form 1"

[0482] One embodiment of the present invention includes an emotion engine in which a generative AI recognizes the emotions of the person who lost their item. Specifically, if the person who lost their item makes an inquiry saying, "I forgot my important umbrella yesterday," the emotion engine recognizes from the tone of voice and expression of the person that they are disappointed. It then conveys this information to the generative AI. Based on this information, the generative AI generates a response that will encourage the person who lost their item. For example, it might generate a response such as, "I'm sorry you forgot your important umbrella. Don't worry, we will do our best to find it for you."

[0483] "Example of form 2"

[0484] In another embodiment of the present invention, voice AI and image AI identify a lost item based on the owner's emotions. Specifically, the voice AI and image AI recognize the owner's anxiety from the tone of voice when the owner says "Find this" and from the accompanying image of the umbrella. Based on this information, the voice AI and image AI prioritize the search for the umbrella. For example, they search the database faster than usual and provide the owner with the results quickly.

[0485] The following describes the processing flow for each example of the form.

[0486] "Example of form 1"

[0487] Step 1: The owner of the lost item makes an inquiry saying, "I left my important umbrella behind yesterday."

[0488] Step 2: The emotional engine recognizes the owner's disappointment from the tone of their voice and expressions.

[0489] Step 3: The emotion engine transmits that information to the generative AI.

[0490] Step 4: The generative AI generates a response that will encourage the owner of the lost item, based on that information.

[0491] "Example of form 2"

[0492] Step 1: Provide a recording of the owner's voice saying "Please find this" along with an image of the umbrella provided.

[0493] Step 2: The voice AI and image AI recognize that the owner is panicking based on the tone of their voice and the image.

[0494] Step 3: The voice AI and image AI use that information to prioritize the search for umbrellas.

[0495] Step 4: Voice AI and image AI search the database faster than usual and quickly provide results to the owner.

[0496] (Example 1)

[0497] Next, we will describe Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0498] In inquiries regarding lost and found items, there is a need to improve user satisfaction by generating appropriate responses that take into account the user's feelings and enabling searches using voice and images. Conventional systems do not adequately consider the user's feelings when generating responses or by using voice and images for searches, which is a challenge.

[0499] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0500] In this invention, the server includes means for combining generative artificial intelligence and a lost and found management device, means for responding to inquiries from users in natural language, and means for analyzing the user's emotions using an emotion recognition device. This enables the generation of responses that take the user's emotions into consideration, and the retrieval of lost items using voice and images.

[0501] "Generative artificial intelligence" is an artificial intelligence technology that uses natural language processing techniques to analyze user inquiries and generate appropriate responses.

[0502] A "lost and found management device" is a device used to manage, search for, and provide information related to lost and found items.

[0503] "User" refers to an individual or organization that makes an inquiry regarding lost and found items.

[0504] "Natural language queries" refer to questions and requests made by users using everyday language.

[0505] An "emotion recognition device" is a device that analyzes and recognizes emotions from the tone and expression of a user's voice.

[0506] A "speech processing device" is a device used to analyze speech data and extract information.

[0507] An "image processing device" is a device used to analyze image data and extract information.

[0508] This invention uses a system that combines generative artificial intelligence and a lost and found management device to respond to user inquiries in natural language. The server utilizes generative artificial intelligence to analyze user inquiries. Specifically, it uses natural language processing technology to extract important information from the inquiry content. In this process, generative AI models such as OpenAI's GPT model can be used.

[0509] Based on the analysis results, the server uses a lost and found management device to search for information on the relevant lost item. The lost and found management device accesses the database and provides information about the lost item. Furthermore, the server uses an emotion recognition device to analyze the user's emotions. The emotion recognition device recognizes emotions from the tone and expression of the user's voice and transmits that information to the generative artificial intelligence.

[0510] Generative artificial intelligence generates encouraging responses to users based on recognized emotional information. For example, if a user inquires, "I forgot my important umbrella yesterday," the server will generate a response such as, "You forgot your important umbrella. Don't worry, we will do our best to find it for you."

[0511] For example, if a user inquires, "I forgot my red umbrella yesterday," the server will provide information about the "red umbrella" that was found "yesterday." Another example of a prompt message would be, "I forgot my red umbrella yesterday, has it been found?"

[0512] The flow of the specific processing in Example 1 will be explained using Figure 15.

[0513] Step 1:

[0514] A user uses a terminal to inquire about a lost item. The user inputs a question in natural language, such as "I left my red umbrella yesterday." The terminal then sends this inquiry to the server.

[0515] Step 2:

[0516] The server receives a query from the user. The server receives a natural language query sent from the terminal as input. The server uses a generative AI model to analyze the query. Specifically, it uses natural language processing techniques to extract important information such as "yesterday" and "red umbrella" from the query. The server then outputs the analyzed information.

[0517] Step 3:

[0518] The server utilizes the lost and found management system based on the analysis results. The input is information analyzed by a generating AI model. The server accesses the lost and found management system's database and searches for information on a "red umbrella" found "yesterday." The output is the corresponding lost and found information.

[0519] Step 4:

[0520] The server uses an emotion recognition device to analyze the user's emotions. The input includes the user's inquiry and tone of voice. Based on this information, the emotion recognition device analyzes the user's emotions and recognizes that they are disappointed. The output is the recognized emotion information.

[0521] Step 5:

[0522] The server uses a generative AI model to generate a response to encourage the user. It uses lost item information and emotional information as input. Based on this information, the generative AI model generates a response such as, "You've forgotten your important umbrella. Don't worry, we'll do our best to find it." The generated response is obtained as output.

[0523] Step 6:

[0524] The server sends the generated response to the user's terminal. The response generated by the generative AI model is used as input. The terminal displays this response to the user, allowing the user to verify it. The output is the response provided to the user.

[0525] (Application Example 1)

[0526] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0527] In modern brick-and-mortar stores, customers frequently leave items behind, requiring prompt and appropriate responses. However, traditional lost and found management systems struggle to address customer feelings, making improving customer satisfaction a challenge.

[0528] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0529] In this invention, the server includes means for combining a generative information processing device and a lost and found management device, means for responding to inquiries from users in natural language, and means for analyzing the user's emotions using an emotion recognition device. This makes it possible to provide lost and found information quickly and appropriately while taking into consideration the customer's emotions.

[0530] A "generative information processing device" is an information processing device that analyzes natural language queries from users and generates appropriate responses.

[0531] A "lost and found item management device" is a device that manages information related to lost and found items and provides that information as needed.

[0532] A "natural language query" refers to a question or request made by a user using the language they use on a daily basis.

[0533] A "speech processing device" is a device used to analyze speech data and extract necessary information.

[0534] An "image processing device" is a device used to analyze image data and extract necessary information.

[0535] An "emotion recognition device" is a device that analyzes a user's emotions from their voice and facial expressions and outputs the results.

[0536] "Means for generating responses" refers to means for creating appropriate responses for users based on analyzed information.

[0537] To implement this invention, a server needs to build a system that integrates a generative information processing device, a lost and found management device, a voice processing device, an image processing device, and an emotion recognition device. The server analyzes natural language queries from users using a generative AI model and generates appropriate responses. Specifically, OpenAI's GPT-3 is used as the generative AI model, and Microsoft Azure's Sentiment Analysis API is used for emotion recognition. The lost and found management device manages information about lost items using a database system such as MySQL.

[0538] The terminal functions as a smartphone or an in-store robot, receiving voice and text input from users. A voice processing unit converts voice data into text, and an image processing unit extracts necessary information from image data. This data is sent to a server and analyzed by a generative information processing unit.

[0539] For example, if a user asks the terminal, "I forgot my blue wallet yesterday," the server analyzes this information and searches the lost and found device for information on a "blue wallet" found "yesterday." If the emotion recognition device detects anxiety from the user's voice, the server generates a response such as, "Don't worry, we will do our best to find it."

[0540] An example of a prompt message would be: "A customer is inquiring about a blue wallet they left behind yesterday. Please generate a message to provide lost item information and reassure the customer."

[0541] The flow of a specific process in Application Example 1 will be explained using Figure 16.

[0542] Step 1:

[0543] The user makes a query to the device using natural language. The input is either the user's voice or text, which the device receives. In the case of voice, the voice processing unit converts the voice data into text. The output is the query in text format.

[0544] Step 2:

[0545] The terminal sends the query content in text format to the server. The server uses a generative AI model to analyze the query content. The input is the query content in text format, which the generative AI model analyzes and extracts relevant keywords and context. The output is the analyzed query content.

[0546] Step 3:

[0547] The server searches the lost and found device for the relevant lost and found information based on the analyzed query. The input is the analyzed query, which generates a database query and sends it to the lost and found device. The output is the relevant lost and found information.

[0548] Step 4:

[0549] The server uses an emotion recognition device to analyze the user's emotions. The input is emotion-related data extracted from the user's voice or text, which the emotion recognition device analyzes. The output is the user's emotional state.

[0550] Step 5:

[0551] The server uses a generative AI model to generate appropriate responses based on the information about the found item and the user's emotional state. The input consists of the information about the found item and the user's emotional state, which the generative AI model uses to generate the response. The output is the response message to the user.

[0552] Step 6:

[0553] The server sends the generated response message to the terminal. The terminal displays or audibly communicates this message to the user. The input is the response message, which the terminal communicates to the user. The output is the response information received by the user.

[0554] (Example 2)

[0555] Next, we will describe Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0556] While there is a need to efficiently and quickly identify lost items, conventional systems do not fully utilize voice and image-based searches, nor do they adjust priorities based on user sentiment. As a result, identifying lost items takes time, leading to decreased user satisfaction.

[0557] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0558] In this invention, the server includes means for combining an information processing device and a lost and found management device, means for analyzing voice data and image data from users, and means for enabling voice and image-based searching using a voice analysis device and an image analysis device. This makes it possible to efficiently identify lost and found items using voice and images.

[0559] An "information processing device" is a device used for inputting, analyzing, and outputting data, and has the function of processing audio data and image data.

[0560] A "lost and found management device" is a device that registers and manages information about lost and found items, and allows for searching and provision of that information as needed.

[0561] "User" refers to an individual or organization that inputs voice or images to search for lost and found items.

[0562] A "speech analysis device" is a device that has the function of analyzing speech data and converting it into text data.

[0563] An "image analysis device" is a device that analyzes image data and has the function of identifying objects and features within the image.

[0564] "Voice data" refers to information input by users via voice, and is the subject of analysis by a voice analysis device.

[0565] "Image data" refers to information input by users through images, and is the subject of analysis by image analysis devices.

[0566] "Search priority" is an indicator that shows the importance and urgency of searching for lost items, and is adjusted based on the user's feelings.

[0567] This invention relates to a lost and found management system using a voice analysis device and an image analysis device. The server combines an information processing device and a lost and found management device to analyze voice data and image data from users. Specifically, voice recognition software is used in the voice analysis device, and image recognition software is used in the image analysis device. This enables efficient identification of lost and found items using voice and images.

[0568] Users use devices such as smartphones or computers to input voice and image information to search for lost items. For example, a user might voice-input "I'm looking for a blue umbrella" and simultaneously upload an image of a blue umbrella to their device. The device then sends this data to the server.

[0569] The server uses a speech analysis device to convert speech data into text and analyze the user's request. It also uses an image analysis device to analyze image data and identify objects and features within the image. Based on this, the server searches the lost and found database and provides the user with appropriate lost and found information.

[0570] For example, if a user inputs "I'm looking for a blue umbrella" via voice input and provides an image of a blue umbrella, the server converts the voice into text using a voice analysis device and analyzes the image using an image analysis device. Based on these analysis results, the server searches its database and provides the user with information about blue umbrellas.

[0571] An example of a prompt message might be, "Upload an image of a blue umbrella and say 'I'm looking for a blue umbrella' aloud." In this way, users can efficiently search for lost items by combining voice and images.

[0572] The flow of the specific processing in Example 2 will be explained using Figure 17.

[0573] Step 1:

[0574] The user inputs voice and images into the device to search for the lost item. Specifically, the user speaks into the smartphone's microphone, "I'm looking for a blue umbrella," and simultaneously takes a picture of a blue umbrella with the camera and uploads it. The input consists of voice and image data, which form the basis for the next processing step.

[0575] Step 2:

[0576] The terminal transmits voice and image data entered by the user to the server. The data is encrypted and transmitted over the internet. The input is voice and image data from the terminal, and the output is the transmission of data to the server.

[0577] Step 3:

[0578] The server passes the received audio data to the speech analysis device. The speech analysis device uses speech recognition software to convert the audio to text and analyze the user's request. The input is audio data, and the output is text data. Specifically, the operation involves analyzing the audio waveform and converting it to text.

[0579] Step 4:

[0580] The server passes the received image data to the image analysis device. The image analysis device uses image recognition software to analyze the image and identify objects and features within it. The input is image data, and the output is feature information from the image. Specifically, it performs pattern recognition and feature extraction from the image.

[0581] Step 5:

[0582] The server searches the lost and found management system's database based on information obtained from the voice analysis device and the image analysis device. The input consists of text data and image feature information, and the output is information on matching lost and found items. Specifically, the server executes database queries and retrieves the results.

[0583] Step 6:

[0584] The server sends the search results to the terminal. The terminal displays the search results to the user. The input is the search result data from the server, and the output is the information displayed to the user. Specifically, the process involves formatting the results and displaying them on the screen.

[0585] (Application Example 2)

[0586] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0587] In modern brick-and-mortar stores, there is a need to quickly and accurately locate lost items and provide customers with the necessary information when they lose something on the premises. However, conventional lost and found management systems lack the ability to use voice or image-based searches, and they do not prioritize items based on customer sentiment, making it difficult to respond quickly.

[0588] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0589] In this invention, the server includes means for combining a generative AI and a lost and found management system, means for enabling voice and image searches through a combination of voice AI and image AI, and means for analyzing the user's emotions and adjusting the search priority. This makes it possible for customers to quickly search for lost items using voice and images and to be provided with information prioritized according to their emotions.

[0590] "Generative AI" is an artificial intelligence technology that generates appropriate responses and information based on user input.

[0591] A "Lost and Found Management System" is a system that manages information on lost and found items and provides that information to users as needed.

[0592] "Voice AI" is an artificial intelligence technology that analyzes voice data, converts it into text data, and reads emotions and intentions from voice.

[0593] "Image AI" is an artificial intelligence technology that analyzes image data to recognize objects and extract their features.

[0594] A "portable information device" is a portable information processing device such as a smartphone or tablet.

[0595] "Methods for analyzing user emotions and adjusting search priority" refers to technologies that analyze user emotions from voice and input data and dynamically change the priority of information retrieval based on the results.

[0596] The system for carrying out this invention consists of a personal information terminal and a server. The personal information terminal has an application installed that accepts voice input and image input. This application allows the user to input information about a lost item using voice or images.

[0597] The server uses speech and image AI to convert audio data received from users into text data using the Google Cloud Speech-to-Text API, and analyzes image data using the Google Cloud Vision API. Furthermore, it uses IBM Watson Tone Analyzer to analyze the emotions from the user's voice and adjust search priorities accordingly.

[0598] The analyzed data is processed by generative AI and searches the database of the lost and found management system. This allows users to quickly obtain information about lost items and display the results on their mobile devices.

[0599] As a concrete example, if a user loses their umbrella in a store, they take a picture of the umbrella with their mobile device and use voice input to say, "Please find my umbrella." Possible prompts include, "Please find the umbrella in this picture. I'm in a hurry," or "Please find the umbrella I dropped in the store." The server analyzes these inputs and provides the user with appropriate lost item information.

[0600] The flow of a specific process in Application Example 2 will be explained using Figure 18.

[0601] Step 1:

[0602] The user takes a picture of the lost item using their mobile device and inputs the phrase "Look for my umbrella" via voice. The input image data and voice data are then sent to the server by the application.

[0603] Step 2:

[0604] The server converts the received audio data into text data using the Google Cloud Speech-to-Text API. This process converts the audio data into text-formatted prompts.

[0605] Step 3:

[0606] The server analyzes the received image data using the Google Cloud Vision API. This analysis identifies objects and features within the image, and the results are output as text data.

[0607] Step 4:

[0608] The server uses IBM Watson Tone Analyzer to analyze user emotions from voice data. Based on the analysis results, data is generated to adjust search priorities.

[0609] Step 5:

[0610] The server uses generative AI to search the lost and found management system database based on text data and image analysis results. This search identifies information about the lost item.

[0611] Step 6:

[0612] The server sends the search results to the mobile device. The device displays information about the lost item to the user, enabling a quick response.

[0613] (Other examples)

[0614] Since this is the same as the specific processing described in the other embodiments of the first embodiment above, the explanation will be omitted.

[0615] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0616] The data generation model 58 is a form of so-called generative AI (Artificial Intelligence). One example of the data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0617] Other examples of generative AI include Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are some examples.

[0618] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0619] [Third Embodiment]

[0620] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0621] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0622] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0623] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0624] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0625] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0626] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0627] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0628] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0629] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0630] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0631] Next, the identification process performed by the identification processing unit 290 of the data processing device 12 will be described.

[0632] "Example of form 1"

[0633] The system of the present invention combines a generative AI and a lost and found management system to respond to inquiries from owners of lost items in conversational form. Specifically, the generative AI analyzes the content of the inquiry from the owner, and based on that, the lost and found management system provides appropriate lost and found information. For example, if the owner inquires, "I forgot my red umbrella yesterday," the generative AI analyzes this inquiry, and the lost and found management system provides information about the "red umbrella" that was found "yesterday."

[0634] "Example of form 2"

[0635] Furthermore, the system of the present invention, when combined with voice AI and image AI, enables searching by voice and image. Specifically, when the owner of a lost item provides voice or image as input, the voice AI and image AI analyze it, and based on that, the lost and found management system provides appropriate lost and found information. For example, if the owner provides an image of an umbrella, the image AI analyzes this image, and the lost and found management system provides information about an umbrella that matches the image.

[0636] The following describes the processing flow for each example of the form.

[0637] "Example of form 1"

[0638] Step 1: The owner makes an inquiry in a conversational message. For example, they might say, "I left my red umbrella behind yesterday."

[0639] Step 2: The generative AI analyzes the content of this inquiry. Through this analysis, it extracts the information that the person forgot their "red umbrella" "yesterday".

[0640] Step 3: The lost and found management system provides appropriate lost and found information based on the analysis results. Specifically, it provides information about a "red umbrella" that was found "yesterday".

[0641] "Example of form 2"

[0642] Step 1: The owner provides audio or images as input. For example, the owner provides an image of an umbrella.

[0643] Step 2: Voice AI and image AI analyze the provided audio and images. This analysis extracts the features of the umbrella.

[0644] Step 3: The lost and found management system provides appropriate lost and found information based on the analysis results. Specifically, it provides information about an umbrella that matches the image.

[0645] (Example 1)

[0646] Next, we will describe Embodiment 1 of Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0647] In managing lost and found items, it is essential to respond quickly and accurately to inquiries from owners. However, conventional systems have the problem of being inconvenient for users because they take a long time to analyze the content of inquiries and search for related information. In addition, there is a lack of search functions using voice and images, which prevents them from meeting the diverse needs of users.

[0648] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0649] In this invention, the server includes means for analyzing the content of an inquiry using an information processing device and natural language processing technology, means for searching for relevant information from a database based on the analysis results, and means for providing the search results to the user. This enables the user to quickly and accurately obtain information about lost items. Furthermore, by combining voice processing technology and image processing technology, searches using voice and images become possible, meeting the diverse needs of users.

[0650] An "information processing device" is a device used for inputting, processing, and outputting data, and includes computers and servers.

[0651] "Natural language processing technology" refers to technologies that enable computers to understand, interpret, and generate human language, and utilize text analysis and language models.

[0652] "Generative artificial intelligence" refers to artificial intelligence that has the ability to generate new information or content based on given data or prompts.

[0653] A "lost and found management device" is a device used to manage, search for, and provide information related to lost and found items.

[0654] "Speech processing technology" refers to technologies for analyzing, recognizing, converting, and generating audio data.

[0655] "Image processing technology" refers to the technology used to analyze, recognize, transform, and generate image data.

[0656] A "database" is a system for efficiently storing, searching, and managing data.

[0657] A "user" is an individual or organization that attempts to obtain information using the system.

[0658] This invention is a system that efficiently processes inquiries about lost items using an information processing device. The server utilizes generative artificial intelligence to analyze inquiries from users in natural language. Specifically, the server uses natural language processing technology to extract important keywords, dates, and item characteristics from the inquiry text. Generative AI models such as OpenAI's GPT-4 can be used for this analysis.

[0659] Based on the analysis results, the server queries the lost and found management device and searches the database for relevant lost and found information. The database can efficiently retrieve information using SQL queries and other methods. The search results are provided to the user from the server via a terminal. The user can use the terminal to check detailed information about the lost and found item.

[0660] Furthermore, by combining voice processing and image processing technologies, it becomes possible to perform searches using voice and images. This allows users to make inquiries more intuitively through voice input and image uploads.

[0661] For example, if a user inquires, "I lost my blue wallet last Friday," the server uses a generative AI model to extract the keywords "last Friday" and "blue wallet." It then searches the lost and found system based on this information, and if a matching item is found, it provides the user with detailed information.

[0662] Examples of prompt messages include the following:

[0663] "User input: 'I lost my blue wallet last Friday.'"

[0664] "Generated AI prompt: 'Please search for information about the blue wallet found last Friday.'"

[0665] In this way, users can quickly and accurately obtain information about lost items.

[0666] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0667] Step 1:

[0668] The user uses their device to enter an inquiry about a lost item. The entered text data is sent from the user's device to the server. For example, if the user enters "I forgot my red umbrella yesterday," that text is sent to the server.

[0669] Step 2:

[0670] The server sends the received query to a generative AI model. The input is text data from the user. The server uses natural language processing technology to extract important keywords, dates, and item characteristics from the query. Specifically, the generative AI model analyzes keywords such as "yesterday" and "red umbrella," and obtains these keywords as output.

[0671] Step 3:

[0672] The server queries the lost and found management device based on the analysis results obtained from the generated AI model. The input is the analyzed keywords. The server uses an SQL query to search the database and identify information about a "red umbrella" found "yesterday". The output is detailed information about the corresponding lost item. Specifically, the server retrieves information about the corresponding lost item from the database.

[0673] Step 4:

[0674] The server provides users with information obtained from the lost and found management device. The input is information about lost items retrieved from a database. The server sends this information to the terminal, and the user can view detailed information about the relevant lost item through the terminal. Specifically, the information is displayed on the user's terminal.

[0675] (Application Example 1)

[0676] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0677] In modern brick-and-mortar stores, customers frequently forget items, but responding quickly and accurately to inquiries about lost items presents a challenge. In particular, there is a need to provide appropriate lost and found information in response to inquiries made in natural language.

[0678] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0679] In this invention, the server includes means for combining a generative information processing device and a lost and found management device, means for responding to inquiries from users in natural language, and means for enabling voice and image searches through a combination of a voice processing device and an image processing device. As a result, a user can make an inquiry about a lost item using a mobile information terminal, the generative information processing device will analyze the content of the inquiry, and the lost and found management device will quickly provide the relevant lost and found information.

[0680] A "generative information processing system" is an information processing system that analyzes queries in natural language and generates appropriate information.

[0681] A "lost and found item management device" is a device that manages information about lost and found items and provides the relevant lost and found item information as needed.

[0682] "User" refers to an individual or organization that uses the system to inquire about lost items.

[0683] "Natural language" refers to the language that humans use on a daily basis, and is the linguistic form that a system analyzes.

[0684] A "speech processing device" is a device used to analyze speech data and extract necessary information.

[0685] An "image processing device" is a device used to analyze image data and extract necessary information.

[0686] "Personal information terminals" refer to portable information processing devices such as smartphones and tablets.

[0687] "Inquiry content" refers to the content of questions or requests that users make to the system.

[0688] "Relevant lost and found information" refers to information about lost and found items identified based on the content of the inquiry.

[0689] To implement this invention, it is necessary to construct a system combining a server, a personal information terminal, a generative information processing device, a lost and found management device, a voice processing device, and an image processing device. The server uses the generative information processing device to analyze natural language inquiries from users and generate appropriate lost and found information. The personal information terminal functions as an interface for users to inquire about lost items.

[0690] Specifically, a user might use a mobile device to make an inquiry such as, "I forgot my blue wallet yesterday." The server sends this inquiry to a generative information processing device, which then analyzes the inquiry using a generative AI model. Based on the analysis results, the lost and found management device searches the database and identifies the relevant lost and found item. If voice and image processing devices are also included, searches using voice and image data are also possible.

[0691] This system allows users to quickly and accurately retrieve information about lost items. An example of a prompt message to the generating AI model is: "User inquiry: 'I forgot my blue wallet yesterday.' Extract the necessary information and generate keywords to query the lost and found management system."

[0692] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0693] Step 1:

[0694] A user uses a mobile device to inquire about a lost item. The input is natural language text, for example, "I forgot my blue wallet yesterday." The device sends this text to the server.

[0695] Step 2:

[0696] The server passes the received text to a generative information processing unit. A generative AI model analyzes this text and extracts important keywords (e.g., dates, object characteristics). The output is a list of the extracted keywords.

[0697] Step 3:

[0698] The server sends the extracted keywords to the lost and found device. The lost and found device searches the database and identifies the corresponding lost and found information. The input is a list of keywords, and the output is the corresponding lost and found information.

[0699] Step 4:

[0700] The server sends the information received from the lost and found device back to the mobile device. The device displays the relevant lost and found information to the user. The output is the lost and found information presented to the user.

[0701] Step 5:

[0702] If necessary, audio and image processing devices analyze the audio and image data to provide additional information. The input is audio or image data, and the output is the analyzed information.

[0703] (Example 2)

[0704] Next, we will describe Example 2 of the morphological example. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0705] In managing lost and found items, there is a challenge in that it is difficult for owners to quickly and accurately identify their lost items using voice or images. Furthermore, conventional systems have limitations in voice and image-based searches, resulting in insufficient user convenience.

[0706] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0707] In this invention, the server includes means that combine generative artificial intelligence and a lost and found management device, means that enable searching using voice data and image data through a combination of voice processing artificial intelligence and image processing artificial intelligence, and means that search a database based on extracted feature information to identify lost and found information. This makes it possible for the owner of a lost item to quickly and accurately identify the lost item using voice and images.

[0708] "Generative artificial intelligence" refers to artificial intelligence technology that generates appropriate responses and information based on user input.

[0709] A "lost and found management device" is a system for managing information about lost and found items and for searching and identifying them.

[0710] "Speech processing artificial intelligence" refers to artificial intelligence that possesses the technology to analyze speech data and convert it into text data.

[0711] "Image processing artificial intelligence" refers to artificial intelligence that possesses the technology to analyze image data and extract feature information.

[0712] "Audio data" refers to data that records audio in digital format.

[0713] "Image data" refers to data recorded in digital format.

[0714] "Feature information" refers to information extracted from images or audio to identify specific objects or content.

[0715] A "database" is a digital record system that systematically organizes information and allows for efficient searching and management.

[0716] A description of embodiments for carrying out this invention will be given.

[0717] Users utilize devices such as smartphones and personal computers to access the lost and found management system. Users can input voice or image descriptions of the lost item into their device. For example, a user might take a picture of an umbrella they lost at a train station and upload that image to the system.

[0718] The terminal transmits voice or image data input by the user to the server. The server uses speech processing artificial intelligence to convert the voice data into text data, and uses image processing artificial intelligence to analyze the image data and extract feature information. In this process, general speech recognition software is used for voice processing, and general image analysis software is used for image processing.

[0719] The server searches the database of the lost and found device based on the extracted characteristic information to identify the lost item. This allows users to obtain information about lost items quickly and accurately.

[0720] For example, if a user voice-inputs "I lost my blue wallet," the server uses voice processing artificial intelligence to extract the text "blue wallet" and sends it to the lost and found device. The system searches the database and returns information about the blue wallet to the server. The server provides this information to the user, who can then check how to retrieve their lost item at the lost and found center.

[0721] An example of a prompt message is, "Please analyze the image of the umbrella I lost at the station and provide information about the lost item." By using this prompt message, the user can give specific instructions to the system.

[0722] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0723] Step 1:

[0724] The user inputs audio or images describing the characteristics of the lost item into the device. For example, the user might take a picture of an umbrella they lost at a train station. This input data is saved on the device as audio or image data.

[0725] Step 2:

[0726] The terminal transmits voice or image data entered by the user to the server. A secure protocol is used over the internet for transmission, ensuring that the data reaches the server safely.

[0727] Step 3:

[0728] The server passes the received audio data to an AI for speech processing, which converts it into text data. The AI ​​uses speech recognition technology to analyze the audio data and generate the corresponding text. For example, the audio "I lost my blue wallet" is converted into the text "blue wallet".

[0729] Step 4:

[0730] The server passes the received image data to an image processing artificial intelligence (AI) to extract feature information. The AI ​​uses image analysis techniques to analyze the image data and identify the features of objects. For example, from an image of an umbrella, the feature information "black umbrella" is extracted.

[0731] Step 5:

[0732] The server searches the lost and found management device's database based on text data and feature information obtained from speech processing artificial intelligence and image processing artificial intelligence. The database search identifies the relevant lost and found information.

[0733] Step 6:

[0734] The server provides users with information about lost items retrieved from the database. Users can view detailed information about the lost items on their terminal screen and learn how to retrieve them. This allows users to quickly and accurately identify lost items.

[0735] (Application Example 2)

[0736] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as a "server," and the headset-type terminal 314 will be referred to as a "terminal."

[0737] In physical stores, there is a challenge in quickly and accurately identifying and returning lost or forgotten items to their owners. Traditional methods often involve manual management of lost items, which is time-consuming and labor-intensive, and increases the risk of misidentification and overlooking information. This raises concerns about decreased customer satisfaction and negatively impacting the store's credibility.

[0738] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0739] In this invention, the server includes means that combine a generation system information processing device and a lost and found management device, means that enable searching by voice and image through a combination of a voice processing device and an image processing device, and means that input and analyze information on lost and found items using a portable information terminal or visual information device in a physical store. This enables the rapid and accurate identification and management of lost and found items in a physical store.

[0740] A "generative information processing device" is a device that executes algorithms and models for generating information, and in particular, a device that has the function of generating appropriate information based on user input.

[0741] A "lost and found management device" is a device that has the function of managing, identifying, and tracking information about lost and found items.

[0742] A "speech processing device" is a device that has the function of analyzing speech data and converting it into text data.

[0743] An "image processing device" is a device that has the function of analyzing image data and extracting features.

[0744] A "portable information terminal" is a portable information processing device used by users to input or retrieve information.

[0745] A "visual information device" is a device for acquiring and processing visual information, and in particular, a device that has the function of analyzing images and videos.

[0746] The invention will now be described in terms of embodiments for carrying out the invention. This invention is a system for streamlining the management of lost and found items in physical stores. The server comprises a generation system information processing device, a lost and found item management device, an audio processing device, and an image processing device. This makes it possible to identify and manage lost and found items using audio and images.

[0747] The user takes a picture of the found item or inputs a voice description using a mobile device or visual information device. The device sends this data to the server. The server converts the voice data to text using a voice processing device and analyzes the image data using an image processing device. Based on the analysis results, the found item management device searches the database for matching found item information and provides it to the user.

[0748] As a concrete example, a store staff member takes a picture of an umbrella they find in the store using a mobile device and uploads it to the system. The server uses an image processing device to analyze the umbrella's features, searches the database for matching lost item information, and provides it to the staff member. In this process, it is also possible to generate appropriate answers to user inquiries using a generative AI model.

[0749] An example of a prompt message would be: "Please provide lost and found information that matches this umbrella in the image."

[0750] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0751] Step 1:

[0752] The user takes a picture of the found item using a mobile device or visual information device and inputs a voice description. The input data consists of an image file and an audio file. The device sends this data to the server.

[0753] Step 2:

[0754] The server uses an audio processing unit to convert audio files into text data. The input is an audio file, and the output is text data. This process uses speech recognition technology to analyze the content of the audio and convert it into text information.

[0755] Step 3:

[0756] The server analyzes image files using an image processing device. The input is an image file, and the output is image feature data. This process uses image recognition technology to identify objects within the image and extract their features.

[0757] Step 4:

[0758] The server uses a lost and found management device to search the database for matching lost and found information based on the analyzed text data and image feature data. The input is text data and image feature data, and the output is lost and found information. This process uses database search technology to quickly identify information that matches the input data.

[0759] Step 5:

[0760] The server uses a generative AI model to generate appropriate answers to user inquiries. The input is the user's inquiry, and the output is the generated answer. This process uses natural language processing techniques to understand the user's intent and provide appropriate information.

[0761] Step 6:

[0762] The server sends search results and generated answers to the terminal and provides them to the user. The input is information about the found item and the generated answers, and the output is information provided to the user. This process uses communication technology to quickly deliver the information the user needs.

[0763] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0764] "Example of form 1"

[0765] One embodiment of the present invention includes an emotion engine in which a generative AI recognizes the emotions of the person who lost their item. Specifically, if the person who lost their item makes an inquiry saying, "I forgot my important umbrella yesterday," the emotion engine recognizes from the tone of voice and expression of the person that they are disappointed. It then conveys this information to the generative AI. Based on this information, the generative AI generates a response that will encourage the person who lost their item. For example, it might generate a response such as, "I'm sorry you forgot your important umbrella. Don't worry, we will do our best to find it for you."

[0766] "Example of form 2"

[0767] In another embodiment of the present invention, voice AI and image AI identify a lost item based on the owner's emotions. Specifically, the voice AI and image AI recognize the owner's anxiety from the tone of voice when the owner says "Find this" and from the accompanying image of the umbrella. Based on this information, the voice AI and image AI prioritize the search for the umbrella. For example, they search the database faster than usual and provide the owner with the results quickly.

[0768] The following describes the processing flow for each example of the form.

[0769] "Example of form 1"

[0770] Step 1: The owner of the lost item makes an inquiry saying, "I left my important umbrella behind yesterday."

[0771] Step 2: The emotional engine recognizes the owner's disappointment from the tone of their voice and expressions.

[0772] Step 3: The emotion engine transmits that information to the generative AI.

[0773] Step 4: The generative AI generates a response that will encourage the owner of the lost item, based on that information.

[0774] "Example of form 2"

[0775] Step 1: Provide a recording of the owner's voice saying "Please find this" along with an image of the umbrella provided.

[0776] Step 2: The voice AI and image AI recognize that the owner is panicking based on the tone of their voice and the image.

[0777] Step 3: The voice AI and image AI use that information to prioritize the search for umbrellas.

[0778] Step 4: Voice AI and image AI search the database faster than usual and quickly provide results to the owner.

[0779] (Example 1)

[0780] Next, we will describe Embodiment 1 of Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0781] In inquiries regarding lost and found items, there is a need to improve user satisfaction by generating appropriate responses that take into account the user's feelings and enabling searches using voice and images. Conventional systems do not adequately consider the user's feelings when generating responses or by using voice and images for searches, which is a challenge.

[0782] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0783] In this invention, the server includes means for combining generative artificial intelligence and a lost and found management device, means for responding to inquiries from users in natural language, and means for analyzing the user's emotions using an emotion recognition device. This enables the generation of responses that take the user's emotions into consideration, and the retrieval of lost items using voice and images.

[0784] "Generative artificial intelligence" is an artificial intelligence technology that uses natural language processing techniques to analyze user inquiries and generate appropriate responses.

[0785] A "lost and found management device" is a device used to manage, search for, and provide information related to lost and found items.

[0786] "User" refers to an individual or organization that makes an inquiry regarding lost and found items.

[0787] "Natural language queries" refer to questions and requests made by users using everyday language.

[0788] An "emotion recognition device" is a device that analyzes and recognizes emotions from the tone and expression of a user's voice.

[0789] A "speech processing device" is a device used to analyze speech data and extract information.

[0790] An "image processing device" is a device used to analyze image data and extract information.

[0791] This invention uses a system that combines generative artificial intelligence and a lost and found management device to respond to user inquiries in natural language. The server utilizes generative artificial intelligence to analyze user inquiries. Specifically, it uses natural language processing technology to extract important information from the inquiry content. In this process, generative AI models such as OpenAI's GPT model can be used.

[0792] Based on the analysis results, the server uses a lost and found management device to search for information on the relevant lost item. The lost and found management device accesses the database and provides information about the lost item. Furthermore, the server uses an emotion recognition device to analyze the user's emotions. The emotion recognition device recognizes emotions from the tone and expression of the user's voice and transmits that information to the generative artificial intelligence.

[0793] Generative artificial intelligence generates encouraging responses to users based on recognized emotional information. For example, if a user inquires, "I forgot my important umbrella yesterday," the server will generate a response such as, "You forgot your important umbrella. Don't worry, we will do our best to find it for you."

[0794] For example, if a user inquires, "I forgot my red umbrella yesterday," the server will provide information about the "red umbrella" that was found "yesterday." Another example of a prompt message would be, "I forgot my red umbrella yesterday, has it been found?"

[0795] The flow of the specific processing in Example 1 will be explained using Figure 15.

[0796] Step 1:

[0797] A user uses a terminal to inquire about a lost item. The user inputs a question in natural language, such as "I left my red umbrella yesterday." The terminal then sends this inquiry to the server.

[0798] Step 2:

[0799] The server receives a query from the user. The server receives a natural language query sent from the terminal as input. The server uses a generative AI model to analyze the query. Specifically, it uses natural language processing techniques to extract important information such as "yesterday" and "red umbrella" from the query. The server then outputs the analyzed information.

[0800] Step 3:

[0801] The server utilizes the lost and found management system based on the analysis results. The input is information analyzed by a generating AI model. The server accesses the lost and found management system's database and searches for information on a "red umbrella" found "yesterday." The output is the corresponding lost and found information.

[0802] Step 4:

[0803] The server uses an emotion recognition device to analyze the user's emotions. The input is the user's inquiry and tone of voice. Based on this information, the emotion recognition device analyzes the user's emotions and recognizes that they are disappointed. The output is the recognized emotion information.

[0804] Step 5:

[0805] The server uses a generative AI model to generate a response to encourage the user. It uses lost item information and emotional information as input. Based on this information, the generative AI model generates a response such as, "You've forgotten your important umbrella. Don't worry, we'll do our best to find it." The generated response is obtained as output.

[0806] Step 6:

[0807] The server sends the generated response to the user's terminal. The response generated by the generative AI model is used as input. The terminal displays this response to the user, allowing the user to verify it. The output is the response provided to the user.

[0808] (Application Example 1)

[0809] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0810] In modern brick-and-mortar stores, customers frequently leave items behind, requiring prompt and appropriate responses. However, traditional lost and found management systems struggle to address customer feelings, making it difficult to improve customer satisfaction.

[0811] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0812] In this invention, the server includes means for combining a generative information processing device and a lost and found management device, means for responding to inquiries from users in natural language, and means for analyzing the user's emotions using an emotion recognition device. This makes it possible to provide lost and found information quickly and appropriately while taking into consideration the customer's emotions.

[0813] A "generative information processing device" is an information processing device that analyzes natural language queries from users and generates appropriate responses.

[0814] A "lost and found management device" is a device that manages information related to lost and found items and provides that information as needed.

[0815] A "natural language query" refers to a question or request made by a user using the language they use on a daily basis.

[0816] A "speech processing device" is a device used to analyze speech data and extract necessary information.

[0817] An "image processing device" is a device used to analyze image data and extract necessary information.

[0818] An "emotion recognition device" is a device that analyzes a user's emotions from their voice and facial expressions and outputs the results.

[0819] "Means for generating responses" refers to means for creating appropriate responses for users based on analyzed information.

[0820] To implement this invention, a server needs to build a system that integrates a generative information processing device, a lost and found management device, a voice processing device, an image processing device, and an emotion recognition device. The server analyzes natural language queries from users using a generative AI model and generates appropriate responses. Specifically, OpenAI's GPT-3 is used as the generative AI model, and Microsoft Azure's Sentiment Analysis API is used for emotion recognition. The lost and found management device manages information about lost items using a database system such as MySQL.

[0821] The terminal functions as a smartphone or an in-store robot, receiving voice and text input from users. A voice processing unit converts voice data into text, and an image processing unit extracts necessary information from image data. This data is sent to a server and analyzed by a generative information processing unit.

[0822] For example, if a user asks the terminal, "I forgot my blue wallet yesterday," the server analyzes this information and searches the lost and found device for information on a "blue wallet" found "yesterday." If the emotion recognition device detects anxiety from the user's voice, the server generates a response such as, "Don't worry, we will do our best to find it."

[0823] An example of a prompt message would be: "A customer is inquiring about a blue wallet they left behind yesterday. Please provide lost item information and generate a message to reassure the customer."

[0824] The flow of a specific process in Application Example 1 will be explained using Figure 16.

[0825] Step 1:

[0826] The user makes a query to the device using natural language. The input is either the user's voice or text, which the device receives. In the case of voice, the voice processing unit converts the voice data into text. The output is the query in text format.

[0827] Step 2:

[0828] The terminal sends the query content in text format to the server. The server uses a generative AI model to analyze the query content. The input is the query content in text format, which the generative AI model analyzes and extracts relevant keywords and context. The output is the analyzed query content.

[0829] Step 3:

[0830] The server searches the lost and found device for the relevant lost and found information based on the analyzed query. The input is the analyzed query, which generates a database query and sends it to the lost and found device. The output is the relevant lost and found information.

[0831] Step 4:

[0832] The server uses an emotion recognition device to analyze the user's emotions. The input is emotion-related data extracted from the user's voice or text, which the emotion recognition device analyzes. The output is the user's emotional state.

[0833] Step 5:

[0834] The server uses a generative AI model to generate appropriate responses based on the information about the found item and the user's emotional state. The input consists of the information about the found item and the user's emotional state, which the generative AI model uses to generate the response. The output is the response message to the user.

[0835] Step 6:

[0836] The server sends the generated response message to the terminal. The terminal displays or audibly communicates this message to the user. The input is the response message, which the terminal communicates to the user. The output is the response information received by the user.

[0837] (Example 2)

[0838] Next, we will describe Example 2 of the morphological example. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0839] While there is a need to efficiently and quickly identify lost items, conventional systems do not fully utilize voice and image-based searches, nor do they adjust priorities based on user sentiment. As a result, identifying lost items takes time, leading to decreased user satisfaction.

[0840] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0841] In this invention, the server includes means for combining an information processing device and a lost and found management device, means for analyzing voice data and image data from users, and means for enabling voice and image-based searching using a voice analysis device and an image analysis device. This makes it possible to efficiently identify lost and found items using voice and images.

[0842] An "information processing device" is a device used for inputting, analyzing, and outputting data, and has the function of processing audio data and image data.

[0843] A "lost and found management device" is a device that registers and manages information about lost and found items, and searches and provides this information as needed.

[0844] "User" refers to an individual or organization that inputs voice or images to search for lost and found items.

[0845] A "speech analysis device" is a device that has the function of analyzing speech data and converting it into text data.

[0846] An "image analysis device" is a device that analyzes image data and has the function of identifying objects and features within the image.

[0847] "Voice data" refers to information input by users via voice, and is the subject of analysis by voice analysis devices.

[0848] "Image data" refers to information input by users through images, and is the subject of analysis by image analysis devices.

[0849] "Search priority" is an indicator that shows the importance and urgency of searching for lost items, and is adjusted based on the user's feelings.

[0850] This invention relates to a lost and found management system using a voice analysis device and an image analysis device. The server combines an information processing device and a lost and found management device to analyze voice data and image data from users. Specifically, voice recognition software is used in the voice analysis device, and image recognition software is used in the image analysis device. This enables efficient identification of lost and found items using voice and images.

[0851] Users use devices such as smartphones or computers to input voice and image data to search for lost items. For example, a user might voice-input "I'm looking for a blue umbrella" and simultaneously upload an image of a blue umbrella to their device. The device then sends this data to the server.

[0852] The server uses a speech analysis device to convert speech data into text and analyze the user's request. It also uses an image analysis device to analyze image data and identify objects and features within the image. Based on this, the server searches the lost and found database and provides the user with appropriate lost and found information.

[0853] For example, if a user inputs "I'm looking for a blue umbrella" via voice input and provides an image of a blue umbrella, the server converts the voice into text using a voice analysis device and analyzes the image using an image analysis device. Based on these analysis results, the server searches its database and provides the user with information about blue umbrellas.

[0854] An example of a prompt message might be, "Upload an image of a blue umbrella and say 'I'm looking for a blue umbrella' aloud." In this way, users can efficiently search for lost items by combining voice and images.

[0855] The flow of the specific processing in Example 2 will be explained using Figure 17.

[0856] Step 1:

[0857] The user inputs voice and images into the device to search for the lost item. Specifically, the user speaks into the smartphone's microphone, "I'm looking for a blue umbrella," and simultaneously takes a picture of a blue umbrella with the camera and uploads it. The input consists of voice and image data, which form the basis for the next processing step.

[0858] Step 2:

[0859] The terminal transmits voice and image data entered by the user to the server. The data is encrypted and transmitted over the internet. The input is voice and image data from the terminal, and the output is the transmission of data to the server.

[0860] Step 3:

[0861] The server passes the received audio data to the speech analysis device. The speech analysis device uses speech recognition software to convert the audio to text and analyze the user's request. The input is audio data, and the output is text data. Specifically, the operation involves analyzing the audio waveform and converting it to text.

[0862] Step 4:

[0863] The server passes the received image data to the image analysis device. The image analysis device uses image recognition software to analyze the image and identify objects and features within it. The input is image data, and the output is feature information from the image. Specifically, it performs pattern recognition and feature extraction from the image.

[0864] Step 5:

[0865] The server searches the lost and found management system's database based on information obtained from the voice analysis device and the image analysis device. The input consists of text data and image feature information, and the output is information on matching lost and found items. Specifically, the server executes database queries and retrieves the results.

[0866] Step 6:

[0867] The server sends the search results to the terminal. The terminal displays the search results to the user. The input is the search result data from the server, and the output is the information displayed to the user. Specifically, the process involves formatting the results and displaying them on the screen.

[0868] (Application Example 2)

[0869] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0870] In modern brick-and-mortar stores, there is a need to quickly and accurately locate lost items and provide customers with the necessary information when they lose something on the premises. However, conventional lost and found management systems lack the ability to use voice or image-based searches, and they do not prioritize items based on customer sentiment, making it difficult to respond quickly.

[0871] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0872] In this invention, the server includes means for combining a generative AI and a lost and found management system, means for enabling voice and image searches through a combination of voice AI and image AI, and means for analyzing the user's emotions and adjusting the search priority. This makes it possible for customers to quickly search for lost items using voice and images and to be provided with information prioritized according to their emotions.

[0873] "Generative AI" is an artificial intelligence technology that generates appropriate responses and information based on user input.

[0874] A "Lost and Found Management System" is a system that manages information on lost and found items and provides that information to users as needed.

[0875] "Voice AI" is an artificial intelligence technology that analyzes voice data, converts it into text data, and reads emotions and intentions from voice.

[0876] "Image AI" is an artificial intelligence technology that analyzes image data to recognize objects and extract their features.

[0877] A "portable information device" is a portable information processing device such as a smartphone or tablet.

[0878] "Methods for analyzing user emotions and adjusting search priority" refers to technologies that analyze user emotions from voice and input data and dynamically change the priority of information retrieval based on the results.

[0879] The system for carrying out this invention consists of a personal information terminal and a server. The personal information terminal has an application installed that accepts voice input and image input. This application allows the user to input information about a lost item using voice or images.

[0880] The server uses speech and image AI to convert audio data received from users into text data using the Google Cloud Speech-to-Text API, and analyzes image data using the Google Cloud Vision API. Furthermore, it uses IBM Watson Tone Analyzer to analyze the emotions from the user's voice and adjust search priorities accordingly.

[0881] The analyzed data is processed by generative AI and searches the database of the lost and found management system. This allows users to quickly obtain information about lost items and display the results on their mobile devices.

[0882] As a concrete example, if a user loses their umbrella in a store, they take a picture of the umbrella with their mobile device and use voice input to say, "Please find my umbrella." Possible prompts include, "Please find the umbrella in this picture. I'm in a hurry," or "Please find the umbrella I dropped in the store." The server analyzes these inputs and provides the user with appropriate lost item information.

[0883] The flow of a specific process in Application Example 2 will be explained using Figure 18.

[0884] Step 1:

[0885] The user takes a picture of the lost item using their mobile device and inputs the phrase "Look for my umbrella" via voice. The input image data and voice data are then sent to the server by the application.

[0886] Step 2:

[0887] The server converts the received audio data into text data using the Google Cloud Speech-to-Text API. This process converts the audio data into text-formatted prompts.

[0888] Step 3:

[0889] The server analyzes the received image data using the Google Cloud Vision API. This analysis identifies objects and features within the image, and the results are output as text data.

[0890] Step 4:

[0891] The server uses IBM Watson Tone Analyzer to analyze user emotions from voice data. Based on the analysis results, data is generated to adjust search priorities.

[0892] Step 5:

[0893] The server uses generative AI to search the lost and found management system database based on text data and image analysis results. This search identifies information about the lost item.

[0894] Step 6:

[0895] The server sends the search results to the mobile device. The device displays information about the lost item to the user, enabling a quick response.

[0896] (Other examples)

[0897] Since this is the same as the specific processing described in the other embodiments of the first embodiment above, the explanation will be omitted.

[0898] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0899] The data generation model 58 is a form of so-called generative AI (Artificial Intelligence). One example of the data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0900] Other examples of generative AI include Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are some examples.

[0901] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0902] [Fourth Embodiment]

[0903] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0904] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0905] The data processing device 12 includes a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a “computer” related to the technology of this disclosure. The computer 22 includes a processor 28, RAM 30, and storage 32.

[0906] The processor 28, RAM 30, and storage 32 are connected to the bus 34. The database 24 and communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to the network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0907] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0908] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0909] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0910] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0911] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0912] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0913] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0914] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0915] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0916] Next, the identification process performed by the identification processing unit 290 of the data processing device 12 will be described.

[0917] "Example of form 1"

[0918] The system of the present invention combines a generative AI and a lost and found management system to respond to inquiries from owners of lost items in conversational form. Specifically, the generative AI analyzes the content of the inquiry from the owner, and based on that, the lost and found management system provides appropriate lost and found information. For example, if the owner inquires, "I forgot my red umbrella yesterday," the generative AI analyzes this inquiry, and the lost and found management system provides information about the "red umbrella" that was found "yesterday."

[0919] "Example of form 2"

[0920] Furthermore, the system of the present invention, when combined with voice AI and image AI, enables searching by voice and image. Specifically, when the owner of a lost item provides voice or image as input, the voice AI and image AI analyze it, and based on that, the lost and found management system provides appropriate lost and found information. For example, if the owner provides an image of an umbrella, the image AI analyzes this image, and the lost and found management system provides information about an umbrella that matches the image.

[0921] The following describes the processing flow for each example of the form.

[0922] "Example of form 1"

[0923] Step 1: The owner makes an inquiry in a conversational message. For example, they might say, "I left my red umbrella behind yesterday."

[0924] Step 2: The generative AI analyzes the content of this inquiry. Through this analysis, it extracts the information that the person forgot their "red umbrella" "yesterday".

[0925] Step 3: The lost and found management system provides appropriate lost and found information based on the analysis results. Specifically, it provides information about a "red umbrella" that was found "yesterday".

[0926] "Example of form 2"

[0927] Step 1: The owner provides audio or images as input. For example, the owner provides an image of an umbrella.

[0928] Step 2: Voice AI and image AI analyze the provided audio and images. This analysis extracts the features of the umbrella.

[0929] Step 3: The lost and found management system provides appropriate lost and found information based on the analysis results. Specifically, it provides information about an umbrella that matches the image.

[0930] (Example 1)

[0931] Next, we will describe Embodiment 1 of Example Form 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0932] In managing lost and found items, it is essential to respond quickly and accurately to inquiries from owners. However, conventional systems have the problem of being inconvenient for users because they take a long time to analyze the content of inquiries and search for related information. In addition, there is a lack of search functions using voice and images, which prevents them from meeting the diverse needs of users.

[0933] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0934] In this invention, the server includes means for analyzing the content of an inquiry using an information processing device and natural language processing technology, means for searching for relevant information from a database based on the analysis results, and means for providing the search results to the user. This enables the user to quickly and accurately obtain information about lost items. Furthermore, by combining voice processing technology and image processing technology, searches using voice and images become possible, meeting the diverse needs of users.

[0935] An "information processing device" is a device used for inputting, processing, and outputting data, and includes computers and servers.

[0936] "Natural language processing technology" refers to technologies that enable computers to understand, interpret, and generate human language, and utilize text analysis and language models.

[0937] "Generative artificial intelligence" refers to artificial intelligence that has the ability to generate new information or content based on given data or prompts.

[0938] A "lost and found management device" is a device used to manage, search for, and provide information related to lost and found items.

[0939] "Speech processing technology" refers to technologies for analyzing, recognizing, converting, and generating audio data.

[0940] "Image processing technology" refers to the technology used to analyze, recognize, transform, and generate image data.

[0941] A "database" is a system for efficiently storing, searching, and managing data.

[0942] A "user" is an individual or organization that attempts to obtain information using the system.

[0943] This invention is a system that efficiently processes inquiries about lost items using an information processing device. The server utilizes generative artificial intelligence to analyze inquiries from users in natural language. Specifically, the server uses natural language processing technology to extract important keywords, dates, and item characteristics from the inquiry text. Generative AI models such as OpenAI's GPT-4 can be used for this analysis.

[0944] Based on the analysis results, the server queries the lost and found management device and searches the database for relevant lost and found information. The database can efficiently retrieve information using SQL queries and other methods. The search results are provided to the user from the server via a terminal. The user can use the terminal to check detailed information about the lost and found item.

[0945] Furthermore, by combining voice processing and image processing technologies, it becomes possible to perform searches using voice and images. This allows users to make inquiries more intuitively through voice input and image uploads.

[0946] For example, if a user inquires, "I lost my blue wallet last Friday," the server uses a generative AI model to extract the keywords "last Friday" and "blue wallet." It then searches the lost and found system based on this information, and if a matching item is found, it provides the user with detailed information.

[0947] Examples of prompt messages include the following:

[0948] "User input: 'I lost my blue wallet last Friday.'"

[0949] "Generated AI prompt: 'Please search for information about the blue wallet found last Friday.'"

[0950] In this way, users can quickly and accurately obtain information about lost items.

[0951] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0952] Step 1:

[0953] The user uses their device to enter an inquiry about a lost item. The entered text data is sent from the user's device to the server. For example, if the user enters "I forgot my red umbrella yesterday," that text is sent to the server.

[0954] Step 2:

[0955] The server sends the received query to a generative AI model. The input is text data from the user. The server uses natural language processing technology to extract important keywords, dates, and item characteristics from the query. Specifically, the generative AI model analyzes keywords such as "yesterday" and "red umbrella," and obtains these keywords as output.

[0956] Step 3:

[0957] The server queries the lost and found management device based on the analysis results obtained from the generated AI model. The input is the analyzed keywords. The server uses an SQL query to search the database and identify information about a "red umbrella" found "yesterday". The output is detailed information about the corresponding lost item. Specifically, the server retrieves information about the corresponding lost item from the database.

[0958] Step 4:

[0959] The server provides users with information obtained from the lost and found management device. The input is information about lost items retrieved from a database. The server sends this information to the terminal, and the user can view detailed information about the relevant lost item through the terminal. Specifically, the information is displayed on the user's terminal.

[0960] (Application Example 1)

[0961] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0962] In modern brick-and-mortar stores, customers frequently forget items, but responding quickly and accurately to inquiries about lost items presents a challenge. In particular, there is a need to provide appropriate lost and found information in response to inquiries made in natural language.

[0963] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0964] In this invention, the server includes means for combining a generative information processing device and a lost and found management device, means for responding to inquiries from users in natural language, and means for enabling voice and image searches through a combination of a voice processing device and an image processing device. As a result, a user can make an inquiry about a lost item using a mobile information terminal, the generative information processing device will analyze the content of the inquiry, and the lost and found management device will quickly provide the relevant lost and found information.

[0965] A "generative information processing system" is an information processing system that analyzes queries in natural language and generates appropriate information.

[0966] A "lost and found item management device" is a device that manages information about lost and found items and provides the relevant lost and found item information as needed.

[0967] "User" refers to an individual or organization that uses the system to inquire about lost items.

[0968] "Natural language" refers to the language that humans use on a daily basis, and is the linguistic form that a system analyzes.

[0969] A "speech processing device" is a device used to analyze speech data and extract necessary information.

[0970] An "image processing device" is a device used to analyze image data and extract necessary information.

[0971] "Personal information terminals" refer to portable information processing devices such as smartphones and tablets.

[0972] "Inquiry content" refers to the content of questions or requests that users make to the system.

[0973] "Relevant lost and found information" refers to information about lost and found items identified based on the content of the inquiry.

[0974] To implement this invention, it is necessary to construct a system combining a server, a personal information terminal, a generative information processing device, a lost and found management device, a voice processing device, and an image processing device. The server uses the generative information processing device to analyze natural language inquiries from users and generate appropriate lost and found information. The personal information terminal functions as an interface for users to inquire about lost items.

[0975] Specifically, a user might use a mobile device to make an inquiry such as, "I forgot my blue wallet yesterday." The server sends this inquiry to a generative information processing device, which then analyzes the inquiry using a generative AI model. Based on the analysis results, the lost and found management device searches the database and identifies the relevant lost and found item. If voice and image processing devices are also included, searches using voice and image data are also possible.

[0976] This system allows users to quickly and accurately retrieve information about lost items. An example of a prompt message to the generating AI model is: "User inquiry: 'I forgot my blue wallet yesterday.' Extract the necessary information and generate keywords to query the lost and found management system."

[0977] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0978] Step 1:

[0979] A user uses a mobile device to inquire about a lost item. The input is natural language text, for example, "I forgot my blue wallet yesterday." The device sends this text to the server.

[0980] Step 2:

[0981] The server passes the received text to a generative information processing unit. A generative AI model analyzes this text and extracts important keywords (e.g., dates, object characteristics). The output is a list of the extracted keywords.

[0982] Step 3:

[0983] The server sends the extracted keywords to the lost and found device. The lost and found device searches the database and identifies the corresponding lost and found information. The input is a list of keywords, and the output is the corresponding lost and found information.

[0984] Step 4:

[0985] The server sends the information received from the lost and found device back to the mobile device. The device displays the relevant lost and found information to the user. The output is the lost and found information presented to the user.

[0986] Step 5:

[0987] If necessary, audio and image processing devices analyze the audio and image data to provide additional information. The input is audio or image data, and the output is the analyzed information.

[0988] (Example 2)

[0989] Next, we will describe Example 2 of the morphological example. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0990] In managing lost and found items, there is a challenge in that it is difficult for owners to quickly and accurately identify their lost items using voice or images. Furthermore, conventional systems have limitations in voice and image-based searches, resulting in insufficient user convenience.

[0991] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0992] In this invention, the server includes means that combine generative artificial intelligence and a lost and found management device, means that enable searching using voice data and image data through a combination of voice processing artificial intelligence and image processing artificial intelligence, and means that search a database based on extracted feature information to identify lost and found information. This makes it possible for the owner of a lost item to quickly and accurately identify the lost item using voice and images.

[0993] "Generative artificial intelligence" refers to artificial intelligence technology that generates appropriate responses and information based on user input.

[0994] A "lost and found management device" is a system for managing information about lost and found items and for searching and identifying them.

[0995] "Speech processing artificial intelligence" refers to artificial intelligence that possesses the technology to analyze speech data and convert it into text data.

[0996] "Image processing artificial intelligence" refers to artificial intelligence that possesses the technology to analyze image data and extract feature information.

[0997] "Audio data" refers to data that records audio in digital format.

[0998] "Image data" refers to data recorded in digital format.

[0999] "Feature information" refers to information extracted from images or audio to identify specific objects or content.

[1000] A "database" is a digital record system that systematically organizes information and allows for efficient searching and management.

[1001] A description of embodiments for carrying out this invention will be given.

[1002] Users utilize devices such as smartphones and personal computers to access the lost and found management system. Users can input voice or image descriptions of the lost item into their device. For example, a user might take a picture of an umbrella they lost at a train station and upload that image to the system.

[1003] The terminal transmits voice or image data input by the user to the server. The server uses speech processing artificial intelligence to convert the voice data into text data, and uses image processing artificial intelligence to analyze the image data and extract feature information. In this process, general speech recognition software is used for voice processing, and general image analysis software is used for image processing.

[1004] The server searches the database of the lost and found device based on the extracted characteristic information to identify the lost item. This allows users to obtain information about lost items quickly and accurately.

[1005] For example, if a user voice-inputs "I lost my blue wallet," the server uses voice processing artificial intelligence to extract the text "blue wallet" and sends it to the lost and found device. The system searches the database and returns information about the blue wallet to the server. The server provides this information to the user, who can then check how to retrieve their lost item at the lost and found center.

[1006] An example of a prompt message is, "Please analyze the image of the umbrella I lost at the station and provide information about the lost item." By using this prompt message, the user can give specific instructions to the system.

[1007] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1008] Step 1:

[1009] The user inputs audio or images describing the characteristics of the lost item into the device. For example, the user might take a picture of an umbrella they lost at a train station. This input data is saved on the device as audio or image data.

[1010] Step 2:

[1011] The terminal transmits voice or image data entered by the user to the server. A secure protocol is used over the internet for transmission, ensuring that the data reaches the server safely.

[1012] Step 3:

[1013] The server passes the received audio data to an AI for speech processing, which converts it into text data. The AI ​​uses speech recognition technology to analyze the audio data and generate the corresponding text. For example, the audio "I lost my blue wallet" is converted into the text "blue wallet".

[1014] Step 4:

[1015] The server passes the received image data to an image processing artificial intelligence (AI) to extract feature information. The AI ​​uses image analysis techniques to analyze the image data and identify the features of objects. For example, from an image of an umbrella, the feature information "black umbrella" is extracted.

[1016] Step 5:

[1017] The server searches the lost and found management device's database based on text data and feature information obtained from speech processing artificial intelligence and image processing artificial intelligence. The database search identifies the relevant lost and found information.

[1018] Step 6:

[1019] The server provides users with information about lost items retrieved from the database. Users can view detailed information about the lost items on their terminal screen and learn how to retrieve them. This allows users to quickly and accurately identify lost items.

[1020] (Application Example 2)

[1021] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1022] In physical stores, there is a challenge in quickly and accurately identifying and returning lost or forgotten items to their owners. Traditional methods often involve manual management of lost items, which is time-consuming and labor-intensive, and increases the risk of misidentification and overlooking information. This raises concerns about decreased customer satisfaction and negatively impacting the store's credibility.

[1023] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1024] In this invention, the server includes means that combine a generation system information processing device and a lost and found management device, means that enable searching by voice and image through a combination of a voice processing device and an image processing device, and means that input and analyze information on lost and found items using a portable information terminal or visual information device in a physical store. This enables the rapid and accurate identification and management of lost and found items in a physical store.

[1025] A "generative information processing device" is a device that executes algorithms and models for generating information, and in particular, a device that has the function of generating appropriate information based on user input.

[1026] A "lost and found management device" is a device that has the function of managing, identifying, and tracking information about lost and found items.

[1027] A "speech processing device" is a device that has the function of analyzing speech data and converting it into text data.

[1028] An "image processing device" is a device that has the function of analyzing image data and extracting features.

[1029] A "portable information terminal" is a portable information processing device used by users to input or retrieve information.

[1030] A "visual information device" is a device for acquiring and processing visual information, and in particular, a device that has the function of analyzing images and videos.

[1031] The invention will now be described in terms of embodiments for carrying out the invention. This invention is a system for streamlining the management of lost and found items in physical stores. The server comprises a generation system information processing device, a lost and found item management device, an audio processing device, and an image processing device. This makes it possible to identify and manage lost and found items using audio and images.

[1032] The user takes a picture of the found item or inputs a voice description using a mobile device or visual information device. The device sends this data to the server. The server converts the voice data to text using a voice processing device and analyzes the image data using an image processing device. Based on the analysis results, the found item management device searches the database for matching found item information and provides it to the user.

[1033] As a concrete example, a store staff member takes a picture of an umbrella they find in the store using a mobile device and uploads it to the system. The server uses an image processing device to analyze the umbrella's features, searches the database for matching lost item information, and provides it to the staff member. In this process, it is also possible to generate appropriate answers to user inquiries using a generative AI model.

[1034] An example of a prompt message would be: "Please provide lost and found information that matches this umbrella in the image."

[1035] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1036] Step 1:

[1037] The user takes a picture of the found item using a mobile device or visual information device and inputs a voice description. The input data consists of an image file and an audio file. The device sends this data to the server.

[1038] Step 2:

[1039] The server uses an audio processing unit to convert audio files into text data. The input is an audio file, and the output is text data. This process uses speech recognition technology to analyze the content of the audio and convert it into text information.

[1040] Step 3:

[1041] The server analyzes image files using an image processing device. The input is an image file, and the output is image feature data. This process uses image recognition technology to identify objects within the image and extract their features.

[1042] Step 4:

[1043] The server uses a lost and found management device to search the database for matching lost and found information based on the analyzed text data and image feature data. The input is text data and image feature data, and the output is lost and found information. This process uses database search technology to quickly identify information that matches the input data.

[1044] Step 5:

[1045] The server uses a generative AI model to generate appropriate answers to user inquiries. The input is the user's inquiry, and the output is the generated answer. This process uses natural language processing techniques to understand the user's intent and provide appropriate information.

[1046] Step 6:

[1047] The server sends search results and generated answers to the terminal and provides them to the user. The input is information about the found item and the generated answers, and the output is information provided to the user. This process uses communication technology to quickly deliver the information the user needs.

[1048] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1049] "Example of form 1"

[1050] One embodiment of the present invention includes an emotion engine in which a generative AI recognizes the emotions of the person who lost their item. Specifically, if the person who lost their item makes an inquiry saying, "I forgot my important umbrella yesterday," the emotion engine recognizes from the tone of voice and expression of the person that they are disappointed. It then conveys this information to the generative AI. Based on this information, the generative AI generates a response that will encourage the person who lost their item. For example, it might generate a response such as, "I'm sorry you forgot your important umbrella. Don't worry, we will do our best to find it for you."

[1051] "Example of form 2"

[1052] In another embodiment of the present invention, voice AI and image AI identify a lost item based on the owner's emotions. Specifically, the voice AI and image AI recognize the owner's anxiety from the tone of voice when the owner says "Find this" and from the accompanying image of the umbrella. Based on this information, the voice AI and image AI prioritize the search for the umbrella. For example, they search the database faster than usual and provide the owner with the results quickly.

[1053] The following describes the processing flow for each example of the form.

[1054] "Example of form 1"

[1055] Step 1: The owner of the lost item makes an inquiry saying, "I left my important umbrella behind yesterday."

[1056] Step 2: The emotional engine recognizes the owner's disappointment from the tone of their voice and expressions.

[1057] Step 3: The emotion engine transmits that information to the generative AI.

[1058] Step 4: The generative AI generates a response that will encourage the owner of the lost item, based on that information.

[1059] "Example of form 2"

[1060] Step 1: Provide a recording of the owner's voice saying "Please find this" along with an image of the umbrella provided.

[1061] Step 2: The voice AI and image AI recognize that the owner is panicking based on the tone of their voice and the image.

[1062] Step 3: The voice AI and image AI use that information to prioritize the search for umbrellas.

[1063] Step 4: Voice AI and image AI search the database faster than usual and quickly provide results to the owner.

[1064] (Example 1)

[1065] Next, we will describe Embodiment 1 of Example Form 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1066] In inquiries regarding lost and found items, there is a need to improve user satisfaction by generating appropriate responses that take into account the user's feelings and enabling searches using voice and images. Conventional systems do not adequately consider the user's feelings when generating responses or by using voice and images for searches, which is a challenge.

[1067] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1068] In this invention, the server includes means for combining generative artificial intelligence and a lost and found management device, means for responding to inquiries from users in natural language, and means for analyzing the user's emotions using an emotion recognition device. This enables the generation of responses that take the user's emotions into consideration, and the retrieval of lost items using voice and images.

[1069] "Generative artificial intelligence" is an artificial intelligence technology that uses natural language processing techniques to analyze user inquiries and generate appropriate responses.

[1070] A "lost and found management device" is a device used to manage, search for, and provide information related to lost and found items.

[1071] "User" refers to an individual or organization that makes an inquiry regarding lost and found items.

[1072] "Natural language queries" refer to questions and requests made by users using everyday language.

[1073] An "emotion recognition device" is a device that analyzes and recognizes emotions from the tone and expression of a user's voice.

[1074] A "speech processing device" is a device used to analyze speech data and extract information.

[1075] An "image processing device" is a device used to analyze image data and extract information.

[1076] This invention uses a system that combines generative artificial intelligence and a lost and found management device to respond to user inquiries in natural language. The server utilizes generative artificial intelligence to analyze user inquiries. Specifically, it uses natural language processing technology to extract important information from the inquiry content. In this process, generative AI models such as OpenAI's GPT model can be used.

[1077] Based on the analysis results, the server uses a lost and found management device to search for information on the relevant lost item. The lost and found management device accesses the database and provides information about the lost item. Furthermore, the server uses an emotion recognition device to analyze the user's emotions. The emotion recognition device recognizes emotions from the tone and expression of the user's voice and transmits that information to the generative artificial intelligence.

[1078] Generative artificial intelligence generates encouraging responses to users based on recognized emotional information. For example, if a user inquires, "I forgot my important umbrella yesterday," the server will generate a response such as, "You forgot your important umbrella. Don't worry, we will do our best to find it for you."

[1079] For example, if a user inquires, "I forgot my red umbrella yesterday," the server will provide information about the "red umbrella" that was found "yesterday." Another example of a prompt message would be, "I forgot my red umbrella yesterday, has it been found?"

[1080] The flow of the specific processing in Example 1 will be explained using Figure 15.

[1081] Step 1:

[1082] A user uses a terminal to inquire about a lost item. The user inputs a question in natural language, such as "I left my red umbrella yesterday." The terminal then sends this inquiry to the server.

[1083] Step 2:

[1084] The server receives a query from the user. The server receives a natural language query sent from the terminal as input. The server uses a generative AI model to analyze the query. Specifically, it uses natural language processing techniques to extract important information such as "yesterday" and "red umbrella" from the query. The server then outputs the analyzed information.

[1085] Step 3:

[1086] The server utilizes the lost and found management system based on the analysis results. The input is information analyzed by a generating AI model. The server accesses the lost and found management system's database and searches for information on a "red umbrella" found "yesterday." The output is the corresponding lost and found information.

[1087] Step 4:

[1088] The server uses an emotion recognition device to analyze the user's emotions. The input is the user's inquiry and tone of voice. Based on this information, the emotion recognition device analyzes the user's emotions and recognizes that they are disappointed. The output is the recognized emotion information.

[1089] Step 5:

[1090] The server uses a generative AI model to generate a response to encourage the user. It uses lost item information and emotional information as input. Based on this information, the generative AI model generates a response such as, "You've forgotten your important umbrella. Don't worry, we'll do our best to find it." The generated response is obtained as output.

[1091] Step 6:

[1092] The server sends the generated response to the user's terminal. The response generated by the generative AI model is used as input. The terminal displays this response to the user, allowing the user to verify it. The output is the response provided to the user.

[1093] (Application Example 1)

[1094] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1095] In modern brick-and-mortar stores, customers frequently leave items behind, requiring prompt and appropriate responses. However, traditional lost and found management systems struggle to address customer feelings, making it difficult to improve customer satisfaction.

[1096] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1097] In this invention, the server includes means for combining a generative information processing device and a lost and found management device, means for responding to inquiries from users in natural language, and means for analyzing the user's emotions using an emotion recognition device. This makes it possible to provide lost and found information quickly and appropriately while taking into consideration the customer's emotions.

[1098] A "generative information processing device" is an information processing device that analyzes natural language queries from users and generates appropriate responses.

[1099] A "lost and found management device" is a device that manages information related to lost and found items and provides that information as needed.

[1100] A "natural language query" refers to a question or request made by a user using the language they use on a daily basis.

[1101] A "speech processing device" is a device used to analyze speech data and extract necessary information.

[1102] An "image processing device" is a device used to analyze image data and extract necessary information.

[1103] An "emotion recognition device" is a device that analyzes a user's emotions from their voice and facial expressions and outputs the results.

[1104] "Means for generating responses" refers to means for creating appropriate responses for users based on analyzed information.

[1105] To implement this invention, a server needs to build a system that integrates a generative information processing device, a lost and found management device, a voice processing device, an image processing device, and an emotion recognition device. The server analyzes natural language queries from users using a generative AI model and generates appropriate responses. Specifically, OpenAI's GPT-3 is used as the generative AI model, and Microsoft Azure's Sentiment Analysis API is used for emotion recognition. The lost and found management device manages information about lost items using a database system such as MySQL.

[1106] The terminal functions as a smartphone or an in-store robot, receiving voice and text input from users. A voice processing unit converts voice data into text, and an image processing unit extracts necessary information from image data. This data is sent to a server and analyzed by a generative information processing unit.

[1107] For example, if a user asks the terminal, "I forgot my blue wallet yesterday," the server analyzes this information and searches the lost and found device for information on a "blue wallet" found "yesterday." If the emotion recognition device detects anxiety from the user's voice, the server generates a response such as, "Don't worry, we will do our best to find it."

[1108] An example of a prompt message would be: "A customer is inquiring about a blue wallet they left behind yesterday. Please provide lost item information and generate a message to reassure the customer."

[1109] The flow of a specific process in Application Example 1 will be explained using Figure 16.

[1110] Step 1:

[1111] The user makes a query to the device using natural language. The input is either the user's voice or text, which the device receives. In the case of voice, the voice processing unit converts the voice data into text. The output is the query in text format.

[1112] Step 2:

[1113] The terminal sends the query content in text format to the server. The server uses a generative AI model to analyze the query content. The input is the query content in text format, which the generative AI model analyzes and extracts relevant keywords and context. The output is the analyzed query content.

[1114] Step 3:

[1115] The server searches the lost and found device for the relevant lost and found information based on the analyzed query. The input is the analyzed query, which generates a database query and sends it to the lost and found device. The output is the relevant lost and found information.

[1116] Step 4:

[1117] The server uses an emotion recognition device to analyze the user's emotions. The input is emotion-related data extracted from the user's voice or text, which the emotion recognition device analyzes. The output is the user's emotional state.

[1118] Step 5:

[1119] The server uses a generative AI model to generate appropriate responses based on the information about the found item and the user's emotional state. The input consists of the information about the found item and the user's emotional state, which the generative AI model uses to generate the response. The output is the response message to the user.

[1120] Step 6:

[1121] The server sends the generated response message to the terminal. The terminal displays or audibly communicates this message to the user. The input is the response message, which the terminal communicates to the user. The output is the response information received by the user.

[1122] (Example 2)

[1123] Next, we will describe Example 2 of the morphological example. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1124] While there is a need to efficiently and quickly identify lost items, conventional systems do not fully utilize voice and image-based searches, nor do they adjust priorities based on user sentiment. As a result, identifying lost items takes time, leading to decreased user satisfaction.

[1125] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1126] In this invention, the server includes means for combining an information processing device and a lost and found management device, means for analyzing voice data and image data from users, and means for enabling voice and image-based searching using a voice analysis device and an image analysis device. This makes it possible to efficiently identify lost and found items using voice and images.

[1127] An "information processing device" is a device used for inputting, analyzing, and outputting data, and has the function of processing audio data and image data.

[1128] A "lost and found management device" is a device that registers and manages information about lost and found items, and searches and provides this information as needed.

[1129] "User" refers to an individual or organization that inputs voice or images to search for lost and found items.

[1130] A "speech analysis device" is a device that has the function of analyzing speech data and converting it into text data.

[1131] An "image analysis device" is a device that analyzes image data and has the function of identifying objects and features within the image.

[1132] "Voice data" refers to information input by users via voice, and is the subject of analysis by voice analysis devices.

[1133] "Image data" refers to information input by users through images, and is the subject of analysis by image analysis devices.

[1134] "Search priority" is an indicator that shows the importance and urgency of searching for lost items, and is adjusted based on the user's feelings.

[1135] This invention relates to a lost and found management system using a voice analysis device and an image analysis device. The server combines an information processing device and a lost and found management device to analyze voice data and image data from users. Specifically, voice recognition software is used in the voice analysis device, and image recognition software is used in the image analysis device. This enables efficient identification of lost and found items using voice and images.

[1136] Users use devices such as smartphones or computers to input voice and image data to search for lost items. For example, a user might voice-input "I'm looking for a blue umbrella" and simultaneously upload an image of a blue umbrella to their device. The device then sends this data to the server.

[1137] The server uses a speech analysis device to convert speech data into text and analyze the user's request. It also uses an image analysis device to analyze image data and identify objects and features within the image. Based on this, the server searches the lost and found database and provides the user with appropriate lost and found information.

[1138] For example, if a user inputs "I'm looking for a blue umbrella" via voice input and provides an image of a blue umbrella, the server converts the voice into text using a voice analysis device and analyzes the image using an image analysis device. Based on these analysis results, the server searches its database and provides the user with information about blue umbrellas.

[1139] An example of a prompt message might be, "Upload an image of a blue umbrella and say 'I'm looking for a blue umbrella' in your voice." In this way, users can efficiently search for lost items by combining voice and images.

[1140] The flow of the specific processing in Example 2 will be explained using Figure 17.

[1141] Step 1:

[1142] The user inputs voice and images into the device to search for the lost item. Specifically, the user speaks into the smartphone's microphone, "I'm looking for a blue umbrella," and simultaneously takes a picture of a blue umbrella with the camera and uploads it. The input consists of voice and image data, which form the basis for the next processing step.

[1143] Step 2:

[1144] The terminal transmits voice and image data entered by the user to the server. The data is transmitted encrypted over the internet. The input is voice and image data from the terminal, and the output is the transmission of data to the server.

[1145] Step 3:

[1146] The server passes the received audio data to the speech analysis device. The speech analysis device uses speech recognition software to convert the audio to text and analyze the user's request. The input is audio data, and the output is text data. Specifically, the operation involves analyzing the audio waveform and converting it to text.

[1147] Step 4:

[1148] The server passes the received image data to the image analysis device. The image analysis device uses image recognition software to analyze the image and identify objects and features within it. The input is image data, and the output is feature information from the image. Specifically, it performs pattern recognition and feature extraction from the image.

[1149] Step 5:

[1150] The server searches the lost and found management system's database based on information obtained from the voice analysis device and the image analysis device. The input consists of text data and image feature information, and the output is information on matching lost and found items. Specifically, the server executes database queries and retrieves the results.

[1151] Step 6:

[1152] The server sends the search results to the terminal. The terminal displays the search results to the user. The input is the search result data from the server, and the output is the information displayed to the user. Specifically, the process involves formatting the results and displaying them on the screen.

[1153] (Application Example 2)

[1154] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1155] In modern brick-and-mortar stores, there is a need to quickly and accurately locate lost items and provide customers with the necessary information when they lose something on the premises. However, conventional lost and found management systems lack the ability to use voice or image-based searches, and they do not prioritize items based on customer sentiment, making it difficult to respond quickly.

[1156] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1157] In this invention, the server includes means for combining a generative AI and a lost and found management system, means for enabling voice and image searches through a combination of voice AI and image AI, and means for analyzing the user's emotions and adjusting the search priority. This makes it possible for customers to quickly search for lost items using voice and images and to be provided with information prioritized according to their emotions.

[1158] "Generative AI" is an artificial intelligence technology that generates appropriate responses and information based on user input.

[1159] A "Lost and Found Management System" is a system that manages information on lost and found items and provides that information to users as needed.

[1160] "Voice AI" is an artificial intelligence technology that analyzes voice data, converts it into text data, and reads emotions and intentions from voice.

[1161] "Image AI" is an artificial intelligence technology that analyzes image data to recognize objects and extract their features.

[1162] A "portable information device" is a portable information processing device such as a smartphone or tablet.

[1163] "Methods for analyzing user emotions and adjusting search priority" refers to technologies that analyze user emotions from voice and input data and dynamically change the priority of information retrieval based on the results.

[1164] The system for carrying out this invention consists of a personal information terminal and a server. The personal information terminal has an application installed that accepts voice input and image input. This application allows the user to input information about a lost item using voice or images.

[1165] The server uses speech and image AI to convert audio data received from users into text data using the Google Cloud Speech-to-Text API, and analyzes image data using the Google Cloud Vision API. Furthermore, it uses IBM Watson Tone Analyzer to analyze the emotions from the user's voice and adjust search priorities accordingly.

[1166] The analyzed data is processed by generative AI and searches the database of the lost and found management system. This allows users to quickly obtain information about lost items and display the results on their mobile devices.

[1167] As a concrete example, if a user loses their umbrella in a store, they take a picture of the umbrella with their mobile device and use voice input to say, "Please find my umbrella." Possible prompts include, "Please find the umbrella in this picture. I'm in a hurry," or "Please find the umbrella I dropped in the store." The server analyzes these inputs and provides the user with appropriate lost item information.

[1168] The flow of a specific process in Application Example 2 will be explained using Figure 18.

[1169] Step 1:

[1170] The user takes a picture of the lost item using their mobile device and inputs the phrase "Look for my umbrella" via voice. The input image data and voice data are then sent to the server by the application.

[1171] Step 2:

[1172] The server converts the received audio data into text data using the Google Cloud Speech-to-Text API. This process converts the audio data into text-formatted prompts.

[1173] Step 3:

[1174] The server analyzes the received image data using the Google Cloud Vision API. This analysis identifies objects and features within the image, and the results are output as text data.

[1175] Step 4:

[1176] The server uses IBM Watson Tone Analyzer to analyze user emotions from voice data. Based on the analysis results, data is generated to adjust search priorities.

[1177] Step 5:

[1178] The server uses generative AI to search the lost and found management system database based on text data and image analysis results. This search identifies information about the lost item.

[1179] Step 6:

[1180] The server sends the search results to the mobile device. The device displays information about the lost item to the user, enabling a quick response.

[1181] (Other examples)

[1182] Since this is the same as the specific processing described in the other embodiments of the first embodiment above, the explanation will be omitted.

[1183] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1184] The data generation model 58 is a form of so-called generative AI (Artificial Intelligence). One example of the data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1185] Other examples of generative AI include Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are some examples.

[1186] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1187] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1188] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1189] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1190] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1191] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1192] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1193] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1194] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing method for the specific process may be used, which includes computer 22 and multiple other computers.

[1195] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1196] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1197] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1198] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1199] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1200] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1201] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1202] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1203] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[1204] The following is further disclosed regarding the embodiments described above.

[1205] (Claim 1)

[1206] A system that combines generative AI and a lost and found management system, a means of responding to inquiries from the owner of a lost item in conversational form, and a means of enabling voice and image searches through a combination of voice AI and image AI.

[1207] (Claim 2)

[1208] The system according to claim 1, further comprising means for generating an appropriate response based on the content of the inquiry from the owner of the lost item.

[1209] (Claim 3)

[1210] The system according to claim 1, wherein the voice AI and image AI further include means for identifying a found object based on voice and image.

[1211] (Claim 4)

[1212] The system according to claim 1, wherein the generative AI further includes an emotion engine that recognizes the emotions of the person who lost the item.

[1213] (Claim 5)

[1214] The system according to claim 2, further comprising means for generating an appropriate response based on the emotions of the person who lost the item, using the generation AI.

[1215] (Claim 6)

[1216] The system according to claim 3, further comprising means for identifying a lost item based on the emotions of the owner of the lost item.

[1217] "Example 1"

[1218] (Claim 1)

[1219] A means of analyzing query content using an information processing device and natural language processing technology,

[1220] A means of searching for relevant information from a database based on the analysis results,

[1221] Means of providing search results to users,

[1222] A method combining generative artificial intelligence and a lost and found management device,

[1223] A means of responding to inquiries from owners of lost items via written conversation,

[1224] By combining voice processing technology and image processing technology, a means is available that enables searching using both voice and images.

[1225] A system that includes this.

[1226] (Claim 2)

[1227] The system according to claim 1, further comprising means for generating an appropriate response based on the content of an inquiry from the owner of a lost item.

[1228] (Claim 3)

[1229] The system according to claim 1, wherein the voice processing technology and image processing technology further include means for identifying a found object based on the voice and image.

[1230] "Application Example 1"

[1231] (Claim 1)

[1232] A means that combines a generation-type information processing device and a lost and found management device,

[1233] A means of responding to natural language inquiries from users,

[1234] By combining it with an audio processing device and an image processing device, a means is made that enables searching by voice and image.

[1235] A means for users to inquire about lost items using a mobile device,

[1236] A means by which a generating information processing device analyzes the query content and a lost and found management device provides the corresponding lost and found information,

[1237] A system that includes this.

[1238] (Claim 2)

[1239] The system according to claim 1, wherein the generation system information processing device further includes means for generating an appropriate response based on the content of an inquiry from a user.

[1240] (Claim 3)

[1241] The system according to claim 1, wherein the audio processing device and the image processing device further include means for identifying a found object based on audio and images.

[1242] Example 2

[1243] (Claim 1)

[1244] A method combining generative artificial intelligence and a lost and found management device,

[1245] A means of responding to inquiries from the owner of a lost item using natural language,

[1246] By combining speech processing artificial intelligence and image processing artificial intelligence, a means to enable searching using audio and image data is also available.

[1247] A means of converting audio data into text data,

[1248] A means of analyzing image data and extracting feature information,

[1249] A means for searching a database based on extracted characteristic information to identify lost item information,

[1250] A system that includes this.

[1251] (Claim 2)

[1252] The system according to claim 1, further comprising means for generating an appropriate response based on the content of an inquiry from the owner of the lost item.

[1253] (Claim 3)

[1254] The system according to claim 1, wherein the voice processing artificial intelligence and image processing artificial intelligence further include means for identifying a found object based on voice data and image data.

[1255] "Application Example 2"

[1256] (Claim 1)

[1257] A means that combines a generation-type information processing device and a lost and found management device,

[1258] A means of responding to inquiries from the owner of a lost item via text message,

[1259] By combining it with an audio processing device and an image processing device, a means is made that enables searching by voice and image.

[1260] A means of inputting and analyzing information on lost items using a mobile information terminal or visual information device in a physical store,

[1261] A means for providing information on found items based on the analysis results,

[1262] A system that includes this.

[1263] (Claim 2)

[1264] The system according to claim 1, wherein the generation system information processing device further includes means for generating an appropriate response based on the content of the inquiry from the owner of the lost item.

[1265] (Claim 3)

[1266] The system according to claim 1, wherein the audio processing device and the image processing device further include means for identifying a found object based on audio and images.

[1267] "Example 1 of combining an emotion engine"

[1268] (Claim 1)

[1269] A method combining generative artificial intelligence and a lost and found management device,

[1270] A means of responding to natural language inquiries from users,

[1271] A means of analyzing a user's emotions using an emotion recognition device,

[1272] A means for generating a response to encourage the user based on the analysis results,

[1273] By combining it with an audio processing device and an image processing device, a means is made that enables searching by voice and image.

[1274] A system that includes this.

[1275] (Claim 2)

[1276] The system according to claim 1, further comprising means for generating an appropriate response based on the content of an inquiry from a user.

[1277] (Claim 3)

[1278] The system according to claim 1, wherein the audio processing device and the image processing device further include means for identifying a found object based on audio and images.

[1279] "Application example 1 of combining emotional engines"

[1280] (Claim 1)

[1281] A means that combines a generation-type information processing device and a lost and found management device,

[1282] A means of responding to natural language inquiries from users,

[1283] By combining it with an audio processing device and an image processing device, a means is made that enables searching by voice and image.

[1284] A means of analyzing a user's emotions using an emotion recognition device,

[1285] A means for generating a reassuring response to the user based on the analysis results,

[1286] A system that includes this.

[1287] (Claim 2)

[1288] The system according to claim 1, wherein the generation system information processing device further includes means for generating an appropriate response based on the content of an inquiry from a user.

[1289] (Claim 3)

[1290] The system according to claim 1, wherein the audio processing device and the image processing device further include means for identifying a found object based on audio and images.

[1291] "Example 2 of combining an emotion engine"

[1292] (Claim 1)

[1293] A means that combines an information processing device and a lost and found management device,

[1294] A means for analyzing voice and image data from users,

[1295] A means for enabling searching by voice and image using a voice analysis device and an image analysis device,

[1296] A means for identifying lost items based on audio data and image data,

[1297] A means of analyzing user sentiment and adjusting search priority,

[1298] A system that includes this.

[1299] (Claim 2)

[1300] The system according to claim 1, wherein the information processing device further includes means for providing appropriate information based on the content of an inquiry from a user.

[1301] (Claim 3)

[1302] The system according to claim 1, wherein the voice analysis device and the image analysis device further include means for identifying a found object based on voice and images.

[1303] "Application example 2 when combining with an emotional engine"

[1304] (Claim 1)

[1305] A method that combines generative AI and a lost and found management system,

[1306] A means of responding to inquiries from the owner of a lost item via text message,

[1307] By combining voice AI and image AI, a method is available that enables searching using both voice and images.

[1308] A means of analyzing user sentiment and adjusting search priority,

[1309] A means of providing lost item information via an application installed on a mobile device,

[1310] A system that includes this.

[1311] (Claim 2)

[1312] The system according to claim 1, further comprising means for generating an appropriate response based on the content of the inquiry from the owner of the lost item.

[1313] (Claim 3)

[1314] The system according to claim 1, wherein the voice AI and image AI further include means for identifying a found object based on voice and image. [Explanation of Symbols]

[1315] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

[Claim 1] A system comprising a server and a user terminal, The processor of the aforementioned server is The system obtains inquiries from the user terminal, including voice data and image data requesting the search for lost items. Using a speech analysis device, the speech data is converted into text data, and using an image analysis device, the image data is analyzed to extract feature information. Using an emotion engine, the audio data and image data are analyzed. Based on the aforementioned text data and characteristic information, the database of the lost and found management device is searched to identify the lost and found information. The identified lost item information is transmitted to the user terminal in response. If, as a result of the analysis of the voice data and image data by the emotion engine, it is determined from the tone of voice in the voice data and the image data that the user is anxious, the search priority is increased and the database is searched. If, as a result of the analysis of the voice data and image data by the emotion engine, it is determined from the tone and expression of the voice in the voice data that the user is disappointed, the generation AI model is used to generate an encouraging response for the user and transmit it to the user terminal. A system that, if the emotion engine determines that the user is feeling anxious as a result of analyzing the voice data and image data, generates a reassuring response using the generation AI model and transmits it to the user's terminal.

Images