System

Abstract

A system is provided.SOLUTION: The system includes a user who takes a photograph, a means for collecting metadata of the taken photograph, a means for classifying the photograph for each event or season on the basis of the collected metadata, and a means for automatically creating the classified photograph as an album.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, with the spread of smartphones, users take a large number of photos on a daily basis. However, properly organizing and managing these photos can be difficult, resulting in problems such as insufficient photo library space and time-consuming search for specific photos. This requires users to expend a lot of time managing and searching for photos, preventing effective use of photos. The purpose of this invention is to solve these problems and provide an environment in which users can efficiently and easily organize, manage, and search for photos. [Means for solving the problem]

[0005] The present invention solves the above problems by the following means: First, a system is provided that includes a means for collecting users who have taken photos and metadata of the photos, a means for classifying the photos by event or season based on the collected metadata, and a means for automatically creating an album of the classified photos.

[0006] Furthermore, in order to identify similar photos, the system provides a means for extracting the feature values ​​of the photos, a means for calculating the similarity based on the extracted feature values, a means for clustering photos with high similarity, a means for presenting photos that are candidates for deletion from the clustering results, and a means for deleting unnecessary photos after user confirmation.

[0007] In addition, the system provides a means for the user to input or voice a search query using natural language, a means for analyzing the input query, a means for searching for relevant photos in the photo library based on the analysis results, and a means for displaying the search results to the user.

[0008] By combining these methods, users can organize, manage, and search for photos efficiently and easily by automatically categorizing photos immediately after taking them, suggesting the deletion of unnecessary photos, and allowing them to quickly search for specific photos in a natural, conversational way.

[0009] "Photo" refers to a still image taken by a user with a camera.

[0010] "Metadata" is information related to a photo, including the date and time the photo was taken, the location where the photo was taken (GPS information), camera settings, and other data.

[0011] "Classification" refers to grouping photos according to specific criteria based on the photo's metadata.

[0012] An "album" is a folder or collection of classified photos, organized around a particular event or theme.

[0013] "Features" are numerical or vector representations of the content of a photograph, and include data for analyzing photographs in the form of facial recognition, hue histograms, etc.

[0014] "Similarity" refers to a measure of how similar two or more photographs are to each other based on the features of those photographs.

[0015] "Clustering" refers to the process of grouping highly similar photos together.

[0016] "Candidates for deletion" is a list of similar photos that may be deemed unnecessary.

[0017] "Search query" refers to the natural language text or voice command entered by a user to search for a particular photo.

[0018] "Analysis" refers to the process of understanding the entered search query and deriving appropriate search terms.

[0019] The term "system" refers to a collection of hardware and software configured to realize the above means. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] The present invention is a photo management AI system that efficiently organizes, manages, and searches large numbers of photos taken by users. Specific embodiments of each function are described below.

[0042] Auto-organization function

[0043] 1. When a device takes a photo, it collects its metadata (date, time, location, etc.).

[0044] 2. Photos are classified by event or season based on the metadata collected by the device. The classification algorithm analyzes timestamps based on the date and time of the photo and location based on GPS data.

[0045] 3. Your device will automatically create albums based on the classification results, for example, grouping photos taken during a specific period into a "Summer Vacation 2023" album.

[0046] Similar photo cleaning function

[0047] 1. The device scans the photos in the photo library and extracts features from each photo (e.g., facial recognition, hue histogram, etc.).

[0048] 2. The device calculates the similarity between the photos based on the extracted features, for example, using a face recognition algorithm.

[0049] 3. The server clusters photos with high similarity and presents deletion candidates to the user. Photos that the user deems unnecessary are added to the deletion list.

[0050] Conversational photo search feature

[0051] 1. A user types or speaks a search query in natural language, for example, "Show me photos of birthday parties in 2022."

[0052] 2. The device converts the voice input into text, which is then sent to the server.

[0053] 3. The server analyzes the search query and searches for matching photos. The server sets search criteria based on the metadata and extracted features.

[0054] 4. The server sends the discovered photos to the device, which displays them to the user.

[0055] Examples:

[0056] Scenario 1: A user returns from a trip and wants to organize their photos.

[0057] 1. The app automatically organizes 100 photos taken by the user on their smartphone while traveling.

[0058] 2. The device creates an album called "Summer Travel 2023" based on the metadata.

[0059] 3. The device scans the photo library, identifies 15 similar photos (the same scene taken from different angles), and presents them as candidates for deletion.

[0060] 4. The user confirms the deletion and the similar photos are deleted.

[0061] Scenario 2: A user wants to show a particular photo to a friend.

[0062] 1. The user says, "Show me pictures of the beach from last summer's vacation."

[0063] 2. The device converts the speech into text and sends it to the server.

[0064] 3. The server analyzes the query and searches for matching photos.

[0065] 4. The server finds the corresponding photo and sends it to the device.

[0066] 5. The device displays the search results to the user.

[0067] As described above, the system of the present invention significantly improves the efficiency of the user's photo management work, and enables the user to instantly find the photos they need.

[0068] The processing flow will be explained below.

[0069] Auto-organization process steps

[0070] Step 1:

[0071] The user takes a photo.

[0072] Use the camera function of your smartphone to capture a specific scene.

[0073] Step 2:

[0074] The device collects the photo's metadata.

[0075] Obtains the shooting date and time, shooting location (GPS information), camera settings (exposure, ISO, etc.), and scene information.

[0076] Step 3:

[0077] Your device will categorize your photos based on their metadata.

[0078] Analyze photos by event or season based on the date and time of the photo. For example, photos taken within a specific period of time can be classified as "summer events."

[0079] Step 4:

[0080] The device will automatically create albums with categorized photos.

[0081] Create an album called something like "Summer Trip 2023" and add relevant photos to it.

[0082] Similar Photo Cleanup Processing Steps

[0083] Step 1:

[0084] Your device will scan your photo library.

[0085] Load all the photo files and prepare them.

[0086] Step 2:

[0087] The device extracts features from each photo.

[0088] Features are calculated using algorithms such as face recognition, color histogram, and composition analysis.

[0089] Step 3:

[0090] The server calculates the similarity based on the extracted features.

[0091] The cosine similarity and Euclidean distance between feature vectors are calculated to evaluate which photos are similar.

[0092] Step 4:

[0093] The server clusters photos with high similarity.

[0094] Photos with high similarity are grouped into the same cluster.

[0095] Step 5:

[0096] The server presents the deletion candidate list to the user.

[0097] It displays a list of the results and allows the user to choose which photos to delete.

[0098] Step 6:

[0099] The server deletes unwanted photos based on the user's selection.

[0100] Delete unwanted photos from your photo library to free up storage space.

[0101] Conversational Photo Search Processing Steps

[0102] Step 1:

[0103] The user types or speaks a photo search query in natural language.

[0104] For example, type or say, "Show me pictures of Okinawa from last summer vacation."

[0105] Step 2:

[0106] The device converts the speech into text (in the case of voice input).

[0107] A speech recognition algorithm is used to convert instructions into text.

[0108] Step 3:

[0109] The device sends a text query to the server.

[0110] The textual query is sent to the server to make a search request.

[0111] Step 4:

[0112] The server parses the query.

[0113] It uses natural language processing models to interpret the intent of the query and set appropriate search criteria.

[0114] Step 5:

[0115] The server searches the photo library.

[0116] Find photos in your photo library based on the search criteria you set.

[0117] Step 6:

[0118] The server sends the search results to the terminal.

[0119] Sends a list of found photos to your device.

[0120] Step 7:

[0121] The terminal displays the search results to the user.

[0122] Display search results in a gallery view so users can browse photos.

[0123] The above is the specific processing flow for each function of the photo management AI system.

[0124] Example 1

[0125] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0126] In today's world, people take a huge number of photos using smartphones and digital cameras, but it is extremely difficult to efficiently organize these photos and quickly search for the ones they need. Furthermore, when there are a large number of similar photos, the task of reviewing and deleting them one by one takes time and effort. To solve these issues, a system is needed that can automatically organize and manage photos and easily delete similar photos.

[0127] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0128] In this invention, the server includes a means for collecting metadata of captured photos, a means for classifying photos by event or season based on the collected metadata, and a means for automatically creating albums of the classified photos. This allows for efficient organization of photos and quick search for specific photos. The server also includes a means for extracting photo features, a means for calculating similarity based on the extracted features, a means for clustering highly similar photos, a means for presenting photos that are candidates for deletion based on the clustering results, and a means for deleting unnecessary photos. This allows for efficient identification of similar photos and easy deletion of unnecessary photos.

[0129] The "user who took the photo" is a user who uses the camera function to acquire a still image.

[0130] A "metadata collection means" is a device or software that has the ability to automatically obtain additional information associated with a photograph, such as the date and time the photograph was taken, the location, and camera settings.

[0131] A "means for categorizing photos by event or season" is a device or software that has algorithms or functionality that analyzes collected metadata and groups photos based on specific times or events.

[0132] The "means for automatically creating an album of classified photographs" refers to a device or software that has the function of organizing photographs based on the classification results and arranging them into an album format that is easy for the user to view and manage.

[0133] A "means for extracting features" is a device or software that has the function of analyzing important visual features such as facial recognition and hue histograms in photographs and extracting them as numerical data.

[0134] The "means for calculating similarity" is a device or software that has the function of calculating and comparing the similarity between different photographs based on the extracted feature amounts.

[0135] The "clustering means" is a device or software that has an algorithm or function for grouping photos based on the calculated similarity and grouping photos that belong to the same group together.

[0136] The "means for presenting photos as candidates for deletion" refers to a device or software that has the function of displaying to the user photos that may be deemed unnecessary from among the highly similar photos identified by clustering, and suggesting that they be deleted.

[0137] A "means for entering search queries or providing voice commands using natural language" is a device or software that has an interface that allows a user to enter spoken commands to search for photos, or that has the ability to convert voice input into text.

[0138] A "means for searching for relevant photos within a photo library" is a device or software that has the function of identifying matching photos within a photo library using metadata and features based on an analyzed search query.

[0139] The "means for displaying search results to the user" refers to a device or software that has the function of displaying the searched photographs in a form that can be viewed by the user.

[0140] This invention relates to a photo management AI system for efficiently organizing, managing, and searching large numbers of photos taken by users. This system is mainly composed of a terminal and a server, and specific embodiments thereof are described in detail below.

[0141] Auto-organization function

[0142] When a device takes a photo, its metadata (such as the date and time the photo was taken and the location) is automatically collected. This collection is done using the device's camera and GPS functions. Based on the collected metadata, the device analyzes the timestamp and GPS data to classify the photos by event or season. Categorized photos are identified by a classification algorithm within the system, and the device automatically creates an album. For example, an album called "Summer Vacation 2023" is automatically generated.

[0143] Similar photo cleaning function

[0144] The device scans the photos in the photo library and extracts features such as facial recognition and hue histograms. Image processing software, for example, is used to extract the features. The device then calculates the similarity between photos based on the extracted features. A numerical data comparison algorithm is used to calculate this similarity. The server then clusters photos with high similarity and presents them as candidates for deletion. The server also manages the deletion of photos that the user deems unnecessary.

[0145] Conversational photo search feature

[0146] The user types or speaks a search query in natural language. Speech recognition software converts the query into text. The converted text is sent to a server, which analyzes the search query. The server sets search criteria based on metadata and extracted features and searches for matching photos in the photo library. The searched photos are sent from the server to the device, where they are displayed to the user.

[0147] Specific examples

[0148] For example, after a user returns from a trip, 100 photos taken on a smartphone are automatically organized. The device creates an album called "Summer 2023 Trip" based on the metadata. The device then scans the photo library and identifies 15 similar photos (the same scene taken from different angles) and presents them as candidates for deletion. The user can then confirm the deletion and delete the similar photos.

[0149] Or, if a user wants to show a friend a specific photo, they can say, "Show me photos from the beach last summer." The device converts the speech to text and sends the text to the server. The server analyzes the query, finds photos with relevant metadata, and sends those photos to the device. The device then displays the search results to the user.

[0150] This system significantly improves the efficiency of photo management and allows users to quickly find the photos they need. These specific actions allow users to organize and share their memories more easily, without the hassle of daily photo management.

[0151] Prompt Sentence Examples

[0152] 1. "Organize your summer 2022 travel photos"

[0153] 2. "Show me your Halloween pictures from October."

[0154] 3. "Add the latest family photo to my list for deletion."

[0155] 4. "Looking for photos from last year's Christmas party."

[0156] As described above, the present invention provides a system that supports the effective organization, management, and search of photos in a variety of situations.

[0157] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0158] Auto-organization function

[0159] Step 1:

[0160] When a device takes a photo, it automatically collects metadata such as the date, time and location of the photo.

[0161] Input: The photo taken

[0162] Output: Metadata added to photo files (timestamp, GPS information, etc.)

[0163] How it works: When a smartphone's camera takes a photo, the built-in GPS module and timestamp function retrieve the corresponding metadata and embed it in the photo file.

[0164] Step 2:

[0165] Based on the metadata collected by the device, timestamp analysis and GPS data analysis are performed to classify photos by event or season.

[0166] Input: Photo file with metadata

[0167] Output: Photo files with classification tags

[0168] What it does: Metadata analysis algorithms analyze timestamps and GPS data to add classification tags to photos, such as "Summer 2023 Travel."

[0169] Step 3:

[0170] Based on the classification results, the device will automatically create an album called "Summer Vacation 2023."

[0171] Input: Photo files with classification tags

[0172] Output: Auto-generated album folder

[0173] What it does: The photo management software creates folders based on classification tags and moves or copies relevant photos into those folders.

[0174] Similar photo cleaning function

[0175] Step 1:

[0176] The device scans photos in the photo library and extracts features such as facial recognition and hue histogram.

[0177] Input: All photos in your photo library

[0178] Output: Feature data for each photo

[0179] What it does: The facial recognition algorithm identifies faces in a photo and extracts their features as numerical data. In addition, it calculates a hue histogram and stores that as a feature.

[0180] Step 2:

[0181] The device calculates the similarity between photos based on the extracted features.

[0182] Input: Feature data

[0183] Output: Similarity score between each photo

[0184] What it does: It runs an algorithm that compares numerical data and calculates the similarity between photos using a score between 0 and 1.

[0185] Step 3:

[0186] The server clusters photos with high similarity and presents them as candidates for deletion.

[0187] Input: Similarity score

[0188] Output: List of photos to delete

[0189] What it does: Groups photos based on similarity scores and prompts the user, "Do you want to delete these photos? They're similar."

[0190] Step 4:

[0191] The user confirms the deletion and deletes the unwanted photos.

[0192] Input: List of photos to delete

[0193] Output: A cleaned-up photo library

[0194] Specific behavior: When the user presses the confirmation button, the photo data to be deleted will be erased from the system.

[0195] Conversational photo search feature

[0196] Step 1:

[0197] The user types or speaks a search query in natural language.

[0198] Input: Natural language search query or voice input

[0199] Output: The search query as text

[0200] What it does: Your smartphone's voice recognition software converts your speech into text in real time.

[0201] Step 2:

[0202] The device converts the voice input into text and sends the text to the server.

[0203] Input: Voice input

[0204] Output: The search query converted to text

[0205] Specific operation: The speech recognition API analyzes the speech, converts it into a string, and then sends it to the server.

[0206] Step 3:

[0207] The server analyzes the search query and searches for matching photos.

[0208] Input: Search query converted to text

[0209] Output: A list of photos from the search results

[0210] How it works: Natural language processing algorithms analyze your query and match it with metadata and features in your photo library to find matching photos.

[0211] Step 4:

[0212] The server sends the discovered photos to the terminal, which displays them to the user.

[0213] Input: Search result photo list

[0214] Output: The photo that is displayed to the user

[0215] Specific operation: The server sends the search result photo data to the terminal, and the terminal visually presents the photo to the user through an interface that displays the photo.

[0216] (Application example 1)

[0217] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0218] There is a demand for a method to efficiently organize, manage, and search large volumes of photos in physical stores and other locations. The goal is to solve these issues by providing an automatic organization function that classifies photos by event or season based on their metadata, a function to remove similar photos, and a function to search for photos using natural language or voice input. It is also important to efficiently analyze user input and provide fast search results.

[0219] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0220] In this invention, the server includes: a means for collecting users who have taken photos and metadata of the photos; a means for classifying photos by event or season based on the collected metadata; a means for automatically creating albums of the classified photos; a means for accepting user voice or text input; a means for converting the input voice into text; a means for analyzing the input using an AI model; a means for searching for relevant photos from the photo library based on the analysis results; and a means for displaying search results to the user. This allows users to easily organize and manage large numbers of photos and quickly find the photos they need. It also improves the accuracy of photo searches and reduces the burden on users.

[0221] "Photo metadata" is information related to a photograph that has been taken, and includes attribute information such as the date and time of the photograph and the location where the photograph was taken.

[0222] "Classifying by event or season" is the process of classifying photos according to specific events (e.g., birthdays, Christmas) or seasons (e.g., spring, summer) based on collected metadata.

[0223] "Automatically create albums" is a function that automatically generates albums according to specific themes or categories based on classified photos.

[0224] "Accepting user voice or text input" refers to a means for recognizing user voice instructions or text-entered search queries.

[0225] "Converting input speech to text" is the process of using speech recognition technology to convert user-entered voice instructions into text format.

[0226] An "AI model" refers to an artificial intelligence algorithm that has been trained to perform tasks such as natural language processing and image recognition.

[0227] "Searching for relevant photos in the photo library" refers to the process of searching for related photos from the photos in the library based on the analyzed data.

[0228] "Displaying search results to the user" refers to the function of displaying the searched photos on the user's device.

[0229] "Photo features" are numerical representations of specific patterns or structures within an image, and are information extracted in the form of facial recognition, hue histograms, etc.

[0230] "Calculating similarity" is a process of numerically evaluating the similarity between photos based on the extracted features.

[0231] "Clustering" refers to an algorithm that groups photos that are highly similar.

[0232] "Presenting photos as candidates for deletion" refers to the process of presenting photos that are deemed unnecessary based on the clustering results to the user and asking for confirmation of deletion.

[0233] "Converting proper nouns, proper names, personal names, and company names into generic names" is the process of replacing specific names or names with more abstract and generic expressions.

[0234] "Noun to superordinate representation" is the process of generalizing a detailed noun representation to its superordinate category.

[0235] "Generating prompt sentences" refers to the process of creating appropriate input sentences for a generative AI model in response to a user query or request.

[0236] The present invention is a system that allows users to efficiently organize, manage, and search for large amounts of photos in a physical store. Specific embodiments of each function are described below.

[0237] Auto-organization function

[0238] The server first collects metadata (such as the date and time of the photo taken, location, etc.) from the photos taken by the device. Based on this metadata, the photos are classified by event or season. Classification involves analyzing timestamps and GPS data. Albums are automatically created based on the classified photos. For example, photos taken in a particular season would be grouped together in a "2023 Summer Campaign" album.

[0239] Conversational photo search feature

[0240] Users can provide a photo search query via voice or text input. For example, a search request might be, "Show me photos of last year's Christmas decorations." The device converts the voice input into text, which is sent to the server. The server uses a generative AI model to analyze the query and search the photo library for matching photos. The search results are sent to the device and displayed to the user.

[0241] Similar photo cleaning function

[0242] The server scans the photos in the photo library and extracts the features of each photo (face recognition, hue histogram, etc.). Based on the extracted features, the similarity between photos is calculated. Photos with high similarity are clustered and presented to the user as candidates for deletion. Photos that the user deems unnecessary are added to the deletion list.

[0243] Explanation of program processing

[0244] The functions of this invention are implemented using smartphones, tablets, and computers as hardware, and the following libraries as software:

[0245] OpenCV: Used for image processing and face recognition.

[0246] scikit-learn: Used for similarity calculations and clustering.

[0247] transformers: Used for NLP processing and generative AI models.

[0248] Specific examples

[0249] Automatic organization: When a store runs a new spring product campaign, related photos are automatically organized into an album called "Spring 2023 New Product Campaign."

[0250] Conversational photo search: When a store manager says, "Show me pictures of last year's Halloween display," the system searches for and displays relevant photos.

[0251] Prompt Sentence Examples

[0252] Below is an example of a prompt sentence that is input to the NLP model.

[0253] "Show me a picture of last year's Christmas decorations."

[0254] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0255] Step 1:

[0256] The device takes a photo and collects its metadata (date, time, location, etc.). The collected metadata is sent to the server. The input is the photo and its metadata, and the output is the metadata sent to the server.

[0257] Step 2:

[0258] The server analyzes the collected metadata and classifies each photo by event or season. For example, photos taken during the summer are classified as a "summer album." The input is the metadata, and the output is the classification result.

[0259] Step 3:

[0260] Based on the classification results, the server automatically creates a photo album. For example, an album titled "2023 Summer Campaign" is generated. The input is the classification results, and the output is the generated album.

[0261] Step 4:

[0262] A user inputs a photo search query in natural language, either by voice or text, such as a request like "Show me pictures of last year's Christmas decorations." The input is the user query, and the output is speech or text data.

[0263] Step 5:

[0264] The device converts voice input into text. Using voice recognition technology, the input voice is converted into text. The converted text is sent to the server. The input is voice data, and the output is text data.

[0265] Step 6:

[0266] The server uses a generative AI model to analyze the input text query. The analysis identifies the conditions for understanding the query and identifying relevant photos. The input is text data, and the output is the analysis results.

[0267] Step 7:

[0268] The server searches for relevant photos in the photo library based on the analysis results. For example, a photo of "last year's Christmas decorations" is searched for based on metadata and features. The input is the analysis results, and the output is the relevant photos.

[0269] Step 8:

[0270] The server sends the search results to the device, which then displays them to the user. The user can view the photos displayed on the screen. The input is the search results, and the output is the displayed photos.

[0271] Step 9:

[0272] It scans photos in your photo library and extracts features for each photo (face recognition, hue histogram, etc.). The input is the photos in your photo library, and the output is the extracted features.

[0273] Step 10:

[0274] The server calculates the similarity between photos based on the extracted features. For example, it evaluates the similarity numerically using facial recognition or hue information. The input is the features, and the output is the similarity calculation result.

[0275] Step 11:

[0276] Photos with high similarity are clustered and presented to the user as candidates for deletion. Photos that the user deems unnecessary are added to a deletion list. The input is the similarity calculation result, and the output is a presentation of deletion candidates and a deletion list.

[0277] Step 12:

[0278] The user confirms the deletion and the unwanted photos are removed from the photo library. The input is the deletion list and the output is the updated photo library.

[0279] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0280] This invention combines a photo management AI system with an emotion engine that recognizes user emotions, allowing for efficient organization, management, and search of large numbers of photos taken by users, and providing interactions that respond to the user's emotions. Specific embodiments of each function are described below.

[0281] Auto-organization function

[0282] 1. When a device takes a photo, it collects its metadata (date, time, location, etc.).

[0283] 2. Photos are classified by event or season based on the metadata collected by the device. The classification algorithm analyzes timestamps based on the date and time of the photo and location based on GPS data.

[0284] 3. Your device will automatically create albums based on the classification results. For example, photos taken within a specific period will be grouped into a "Summer Vacation 2023" album.

[0285] Similar photo cleaning function

[0286] 1. The device scans the photos in the photo library and extracts features from each photo (e.g., facial recognition, hue histogram, etc.).

[0287] 2. The device calculates the similarity between the photos based on the extracted features, for example, using a face recognition algorithm.

[0288] 3. The server clusters photos with high similarity and presents deletion candidates to the user. Photos that the user deems unnecessary are added to the deletion list.

[0289] Conversational photo search feature

[0290] 1. A user types or speaks a search query in natural language, for example, "Show me photos of birthday parties in 2022."

[0291] 2. The device converts the voice input into text, which is then sent to the server.

[0292] 3. The server analyzes the search query and searches for matching photos. The server sets search criteria based on the metadata and extracted features.

[0293] 4. The server sends the discovered photos to the device, which displays them to the user.

[0294] Emotion recognition and linking functions using an emotion engine

[0295] 1. The device captures the facial expressions and voice of the user viewing the photo in real time and sends them to the emotion engine.

[0296] 2. The emotion engine analyzes the user's emotional state and identifies emotions such as joy, sadness, surprise, etc.

[0297] 3. The device optimizes the display order of photos according to the user's emotions. For example, if the user is happy, photos of happy memories will be displayed first.

[0298] 4. The emotion engine automatically generates albums based on specific emotional states and selects photos that fit the user's mood. For example, if the user is sad, it will suggest an album of "fun trips."

[0299] Examples:

[0300] Scenario 1: A user returns from a trip and wants to organize their photos.

[0301] 1. The app automatically organizes 100 photos taken by the user on their smartphone while traveling.

[0302] 2. The device creates an album called "Summer Travel 2023" based on the metadata.

[0303] 3. The device scans the photo library, identifies 15 similar photos (the same scene taken from different angles), and presents them as candidates for deletion.

[0304] 4. The user confirms the deletion and the similar photos are deleted.

[0305] Scenario 2: A user wants to show a particular photo to a friend.

[0306] 1. The user says, "Show me pictures of the beach from last summer's vacation."

[0307] 2. The device converts the speech into text and sends it to the server.

[0308] 3. The server analyzes the query and searches for matching photos.

[0309] 4. The server finds the corresponding photo and sends it to the device.

[0310] 5. The device displays the search results to the user.

[0311] Scenario 3: When the user is in a particular emotional state.

[0312] 1. The user smiles while viewing the photo.

[0313] 2. The device sends the user's facial expressions to the emotion engine.

[0314] 3. The emotion engine recognizes the user's emotion as "joy."

[0315] 4. The device will prioritize displaying photos of happy events according to the emotion of "joy."

[0316] 5. The emotion engine automatically generates an album of "happy memories" and suggests them to the user.

[0317] As described above, the system of the present invention significantly improves the efficiency of users' photo management tasks, and by providing interactions that respond to the user's emotions, it realizes a more personalized photo organization, management, and search experience.

[0318] The processing flow will be explained below.

[0319] Auto-organization process steps

[0320] Step 1:

[0321] The user takes a photo.

[0322] Use the camera function of your smartphone to capture a specific scene.

[0323] Step 2:

[0324] The device collects the photo's metadata.

[0325] Obtains the shooting date and time, shooting location (GPS information), camera settings (exposure, ISO, etc.), and scene information.

[0326] Step 3:

[0327] Your device will categorize your photos based on their metadata.

[0328] Analyze photos by event or season based on the date and time of the photo. For example, photos taken within a specific period of time can be classified as "summer events."

[0329] Step 4:

[0330] The device will automatically create albums with categorized photos.

[0331] Create an album called something like "Summer Trip 2023" and add relevant photos to it.

[0332] Similar Photo Cleanup Processing Steps

[0333] Step 1:

[0334] Your device will scan your photo library.

[0335] Load all the photo files and prepare them.

[0336] Step 2:

[0337] The device extracts features from each photo.

[0338] Features are calculated using algorithms such as face recognition, color histogram, and composition analysis.

[0339] Step 3:

[0340] The server calculates the similarity based on the extracted features.

[0341] The cosine similarity and Euclidean distance between feature vectors are calculated to evaluate which photos are similar.

[0342] Step 4:

[0343] The server clusters photos with high similarity.

[0344] Photos with high similarity are grouped into the same cluster.

[0345] Step 5:

[0346] The server presents the deletion candidate list to the user.

[0347] It displays a list of the results and allows the user to choose which photos to delete.

[0348] Step 6:

[0349] The server deletes unwanted photos based on the user's selection.

[0350] Delete unwanted photos from your photo library to free up storage space.

[0351] Conversational Photo Search Processing Steps

[0352] Step 1:

[0353] The user types or speaks a photo search query in natural language.

[0354] For example, type or say, "Show me pictures of Okinawa from last summer vacation."

[0355] Step 2:

[0356] The device converts the speech into text (in the case of voice input).

[0357] A speech recognition algorithm is used to convert instructions into text.

[0358] Step 3:

[0359] The device sends a text query to the server.

[0360] The textual query is sent to the server to make a search request.

[0361] Step 4:

[0362] The server parses the query.

[0363] It uses natural language processing models to interpret the intent of the query and set appropriate search criteria.

[0364] Step 5:

[0365] The server searches the photo library.

[0366] Find photos in your photo library based on the search criteria you set.

[0367] Step 6:

[0368] The server sends the search results to the terminal.

[0369] Sends a list of found photos to your device.

[0370] Step 7:

[0371] The terminal displays the search results to the user.

[0372] Display search results in a gallery view so users can browse photos.

[0373] Processing steps for emotion recognition and linkage functions by the emotion engine

[0374] Step 1:

[0375] The device captures the facial expressions and voice of the user viewing the photos in real time.

[0376] A camera and microphone are used to collect facial expression and voice data of the user.

[0377] Step 2:

[0378] The device sends facial expression data and voice data to the emotion engine.

[0379] The collected data is sent to an emotion engine to analyze the user's emotional state.

[0380] Step 3:

[0381] An emotion engine analyzes the user's emotional state.

[0382] It recognizes the user's emotions by applying algorithms that identify emotions such as joy, sadness, and surprise.

[0383] Step 4:

[0384] The device optimizes the display order of photos according to the user's emotions.

[0385] For example, if the user is happy, photos of happy memories are displayed preferentially.

[0386] Step 5:

[0387] The emotional engine automatically generates albums that correspond to specific emotional states.

[0388] Select photos that match the user's mood and create albums for "happy memories" or "refreshment."

[0389] Step 6:

[0390] The terminal proposes automatically generated albums to the user.

[0391] The app displays an album of "fun trips" and makes suggestions to the user.

[0392] The above is the specific processing flow for each function of the photo management AI system that combines the emotion engine.

[0393] Example 2

[0394] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0395] Conventional photo management systems have the drawback of making it difficult for users to effectively organize, manage, and search large numbers of photos, which is time-consuming. In addition, the photo viewing experience is not personalized based on the user's emotions, which hinders user satisfaction.

[0396] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0397] In this invention, the server includes means for collecting photo metadata, means for classifying photos by event or season, means for automatically creating albums of the classified photos, means for analyzing emotions, means for optimizing the display order of photos based on the emotion analysis results, and means for automatically creating albums according to specific themes based on the emotion analysis. This not only enables efficient organization and management of photos, but also provides a personalized photo viewing experience according to the user's emotions.

[0398] "User" refers to the person taking the photograph and using the system.

[0399] "Metadata" is information related to a photo, including details such as the date and time the photo was taken, the location, and camera settings.

[0400] "Classification" refers to the act of organizing photos into specific events or seasons based on collected metadata.

[0401] An "album" is a group of classified photos, which is automatically created based on a specific theme or period.

[0402] "Emotion analysis means" refers to technology for analyzing and identifying a user's emotional state from data such as facial expressions and voice.

[0403] "Optimizing the display order" refers to the act of adjusting the order of displayed photos based on the results of user sentiment analysis.

[0404] A "themed album" refers to an album that is automatically generated by collecting photos that correspond to a specific emotion or theme based on the results of emotional analysis.

[0405] "Features" are numerically quantified information about the content of a photograph, and refer to data that represents the characteristics of a photograph, such as facial recognition and hue histograms.

[0406] "Similarity" refers to a numerical measure for assessing the content similarity between photographs.

[0407] "Clustering" refers to a technique for grouping similar photos based on extracted features.

[0408] "Candidates for deletion" refers to photos that the user has determined to be unnecessary or that are highly similar and therefore useless.

[0409] "Search query" refers to input instructions, specified by a user in natural language or speech, for searching for a particular photo.

[0410] "Parsing" refers to the process of breaking down the input search query and converting it into appropriate search terms.

[0411] This invention is a photo management AI system that enables efficient organization, management, and search of photos taken by users, and also provides interaction based on the user's emotions. This system collects and analyzes photo metadata, automatically classifies and creates albums, identifies similar photos and suggests deletion options, and optimizes photo display through emotion analysis.

[0412] When a user takes a photo on a device, metadata such as the date and time of the photo and the location are collected. The device then uses the camera API to obtain Exif information and collect the metadata. Based on this metadata, the photos are classified by specific events or seasons. Classification is performed by analyzing the timestamp of the photo and GPS data. This allows an album called, for example, "Summer Trip 2023" to be automatically created.

[0413] Next, the device scans the photos in the photo library and extracts features from each photo. Based on the extracted features, the similarity between photos is calculated. Photos with high similarity are grouped using a clustering algorithm and presented as candidates for deletion. For example, photos of the same scene taken from different angles may be grouped into a specific list.

[0414] With the conversational photo search feature, users enter search queries by voice, and the device converts the speech into text and sends it to the server. The server analyzes the search query, searches for matching photos in the photo library, and displays the results to the user. For example, a user can search for photos using a search query such as "Show me photos of the beach from last summer vacation."

[0415] With the emotion analysis function, the device captures the user's facial expressions and voice in real time and sends them to the emotion engine. The emotion engine uses facial expression recognition algorithms and voice emotion analysis algorithms to analyze the user's emotional state and identify emotions such as joy, sadness, and surprise. This allows the device to optimize the display order of photos according to the user's emotions. For example, if the user smiles while viewing photos, the emotion engine will recognize this as "joy" and prioritize displaying photos of happy events. In addition, the device can automatically generate an album of "happy memories" based on the emotion analysis and suggest them to the user.

[0416] The hardware used includes devices such as smartphones, tablets, and PCs, and the required software includes a camera API, a voice recognition API (such as Google Speech-to-Text API), and an emotion analysis algorithm (such as Face++ or EmoVu). Additionally, server-side search systems such as Elasticsearch and Lucene can be used.

[0417] With the above configuration, this system can significantly improve the efficiency of users' photo management tasks and provide a personalized photo browsing experience that reflects their emotions. Specific examples of prompts are as follows:

[0418] "Describe an automated photo organization system based on a generative AI model."

[0419] "Please explain with some examples how to manage photos using the emotion engine."

[0420] This allows users to easily search for specific photos and enjoy photos that match their emotions.

[0421] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0422] Step 1:

[0423] Taking photos and collecting metadata

[0424] When a user takes a photo, the device collects metadata in Exif format (such as the date and time the photo was taken, the location, and the camera settings).

[0425] Input: Photo data, metadata at the time of shooting

[0426] Output: A data object containing the photo and its metadata.

[0427] Specific operation: When the smartphone's camera application saves a photo, it simultaneously obtains the Exif information associated with the photo and stores it in an internal database.

[0428] Step 2:

[0429] Photo classification based on metadata

[0430] The device analyzes the metadata it collects and categorizes photos by event or season based on the date, time, and location they were taken.

[0431] Input: Photo data including metadata

[0432] Output: A list of classified photo data

[0433] What it does: Your device analyzes the timestamps and runs an algorithm to categorize the photos based on events, such as "Summer 2023" or "a specific trip."

[0434] Step 3:

[0435] Auto-create albums

[0436] The device automatically creates an album based on the classification results.

[0437] Input: A list of classified photo data

[0438] Output: Auto-generated album

[0439] Specific operation: The device creates an album named, for example, "Summer Travel 2023," and adds the corresponding photo data to the album.

[0440] Step 4:

[0441] Photo library scanning and feature extraction

[0442] The device scans photos in the photo library and extracts features such as facial recognition and hue histogram.

[0443] Input: Photo data in the photo library

[0444] Output: Feature data for each photo

[0445] Specific operation: The device uses an image processing library such as OpenCV to extract facial features and color distribution from each photo.

[0446] Step 5:

[0447] Calculating similarity between photos

[0448] The device calculates the similarity between photos based on the features extracted.

[0449] Input: Feature data

[0450] Output: Similarity scores and photo pairs

[0451] Specific operation: The device performs a similarity calculation method (e.g., Euclidean distance) based on face recognition and hue distribution to calculate a similarity score.

[0452] Step 6:

[0453] Clustering and suggesting deletion candidates

[0454] The server clusters photos with high similarity and presents them to the user as candidates for deletion.

[0455] Input: Similarity scores and photo pairs

[0456] Output: Clustering results and removal candidate list

[0457] What it does: The server uses a clustering algorithm such as DBSCAN to group similar photos and displays deletion candidates to the user on the interface.

[0458] Step 7:

[0459] Confirm and execute the deletion

[0460] The user selects unwanted photos from the presented list of photos to delete and confirms the deletion.

[0461] Input: Deletion candidate list and user selection

[0462] Output: Updated photo library

[0463] What happens: When the user presses the Delete button, the selected photo is deleted from the photo library.

[0464] Step 8:

[0465] Entering search queries in natural language

[0466] The user enters a search query by voice.

[0467] Input: Audio data

[0468] Output: Text query

[0469] Specific action: Say to your voice assistant or smart speaker, "Show me photos of the beach from last summer vacation."

[0470] Step 9:

[0471] Speech-to-text

[0472] The device converts the voice input into text and sends it to the server.

[0473] Input: Audio data

[0474] Output: Text data

[0475] What happens: A speech recognition API (e.g., Google Speech-to-Text API) analyzes the audio data and generates corresponding text.

[0476] Step 10:

[0477] Search query analysis and photo search

[0478] The server analyzes the text query and searches for matching photos in the photo library.

[0479] Input: Text query

[0480] Output: List of photo data

[0481] Specific operation: Analyzes queries using a natural language processing model (generative AI model) and searches for photos that match the criteria based on metadata and features.

[0482] Step 11:

[0483] Displaying search results

[0484] The server sends the search results to the terminal, which displays them to the user.

[0485] Input: List of photo data

[0486] Output: Displaying the photo on a user interface

[0487] Specific operation: The retrieved photos are displayed on the device display in album or thumbnail format.

[0488] Step 12:

[0489] Acquiring and sending emotion data

[0490] The device captures the user's facial expressions and voice in real time and sends them to the emotion engine.

[0491] Input: Real-time facial expression data, voice data

[0492] Output: Data for sentiment analysis

[0493] Specific behavior: Uses a camera and microphone to capture the user's facial expressions and voice in real time and send them to the emotion engine.

[0494] Step 13:

[0495] Sentiment Analysis and Identification

[0496] The emotion engine analyzes the user's emotional state and identifies emotions such as joy, sadness, surprise, etc.

[0497] Input: Facial expression data, voice data

[0498] Output: Emotional state label (e.g., happy, sad)

[0499] Specific operation: Identify emotions using facial expression recognition algorithms and voice emotion analysis algorithms.

[0500] Step 14:

[0501] Optimize the display order of photos

[0502] The device optimizes the display order of photos according to the user's emotions.

[0503] Input: Emotional state labels, photo data from the photo library

[0504] Output: A list of photos in an optimized display order

[0505] What it does: Based on your emotional state, it rearranges the photos in your photo library and prioritizes photos that evoke positive emotions.

[0506] Step 15:

[0507] Emotion-based album generation

[0508] The emotion engine automatically generates albums that correspond to specific emotional states and suggests them to the user.

[0509] Input: Emotional state labels, classified photo data

[0510] Output: Thematic albums

[0511] Specific operation: For example, if the user is sad, an album of photos of "happy memories" is automatically generated and displayed to the user.

[0512] (Application example 2)

[0513] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0514] In physical stores, there is a need to make optimal product recommendations in real time based on customer emotions. However, conventional methods have the problem that it is difficult for store staff to directly observe customers' emotions and recommend appropriate products, which takes time and effort. Furthermore, because it is not possible to accurately grasp customers' emotions, the recommended products often do not meet the customer's needs. To solve these problems, a system is needed that can recognize customers' emotions in real time and recommend products based on them.

[0515] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing customer emotions in real time, means for optimizing photos to be displayed based on the emotion recognition, means for providing interactions based on the emotion recognition, and means for suggesting target information based on the emotion recognition. This makes it possible to accurately grasp customer emotions and suggest optimal products based on that.

[0516] The "user who took the photo" refers to the person who captured the still image using a camera.

[0517] "Means for collecting metadata from photographs taken" refers to methods for automatically acquiring and storing relevant information such as the date and time a photograph was taken, the location where it was taken, and camera settings.

[0518] "Means for categorizing photos by event or season based on collected metadata" refers to a method for using acquired metadata information to group photos based on a specific time period or specific location.

[0519] "Means for automatically creating albums of classified photos" refers to a method for aggregating classified photos and organizing them into a collection based on a specific theme or event.

[0520] "Means for recognizing a user's emotions in real time" refers to technology that analyzes biometric information such as a user's facial expressions and voice, and identifies a specific emotional state in real time.

[0521] "Means for optimizing photos to be displayed based on emotion recognition" refers to technology that automatically adjusts the order and content of photos to be displayed according to the user's recognized emotion.

[0522] "Means for providing interactions based on emotion recognition" refers to technology that enables a system to automatically provide appropriate responses and actions in accordance with the user's emotional state.

[0523] "Means for suggesting targeted information based on emotion recognition" refers to technology that analyzes a user's emotional state and, based on this, recommends information on optimal products and services to the user.

[0524] This invention relates to a system that efficiently organizes and manages photos taken by users and suggests products based on the user's emotions. This system is particularly effective in brick-and-mortar stores, where store staff can use smart glasses to grasp customers' emotions in real time and make optimal product suggestions.

[0525] System configuration

[0526] The system of the present invention consists of the following major components:

[0527] 1. Smart Glasses

[0528] The smart glasses are equipped with a camera that captures the faces of customers in the store in real time, and also have a display that provides store staff with the necessary information instantly.

[0529] 2. Server

[0530] The server works in conjunction with an emotion recognition engine to analyze customer emotions and generate a list of product suggestions based on the analysis results.

[0531] 3. Emotion Recognition Engine

[0532] It is a software module that analyzes a customer's facial expressions and voice to identify their emotional state, such as joy, sadness, or surprise.

[0533] 4. Product Proposal Module

[0534] It includes an algorithm to suggest optimal products based on the analysis results of the emotion recognition engine.

[0535] Hardware and software used

[0536] Hardware:

[0537] Smart glasses (e.g. Google Glass, Vuzix Blade)

[0538] In-store camera

[0539] software:

[0540] OpenCV: Capture frames from camera stream

[0541] emotion_recognition_module: Detects customer faces from frames and analyzes emotions

[0542] product_recommendation_module: Search for the best products based on analyzed sentiment data

[0543] How it works

[0544] The server acquires the camera stream using OpenCV and captures frames. From these frames, the emotion recognition engine (emotion_recognition_module) analyzes the customer's facial expressions and identifies their emotional state. Based on the identified emotions, the product recommendation module (product_recommendation_module) generates an appropriate product list. The generated list is displayed on the smart glasses display, and store staff receive instructions on how to recommend the most suitable products to the customer.

[0545] Specific examples

[0546] A specific example of use in a physical store is shown below. For example, consider a case where store staff wear smart glasses to make product recommendations. When a customer enters the display area, the camera installed in the smart glasses captures the person's face and recognizes their emotions in real time. For example, if the customer smiles, the glasses will recognize that emotion as "happiness" and suggest a product that matches that emotion (e.g., a luxury sofa).

[0547] Prompt Sentence Examples

[0548] Below is an example of an input prompt for a generative AI model.

[0549] Recognize faces and estimate emotions. Recommend products based on specific emotions.

[0550] import cv2

[0551] import emotion_recognition_module

[0552] import product_recommendation_module

[0553] Get Camera Stream

[0554] camera = cv2.VideoCapture(0)

[0555] Product information list

[0556] products = [

[0557] {"name": "luxury sofa", "emotion": "joy"},

[0558] {"name": "Relax chair", "emotion": "sadness"},

[0559] {"name": "Energetic Lamp", "emotion": "surprise"}

[0560] ]

[0561] while True:

[0562] Capture frames from the camera stream

[0563] ret, frame = camera.read()

[0564] if not ret:

[0565] break

[0566] Detect faces from frames and analyze emotions

[0567] emotions = emotion_recognition_module.detect_emotion(frame)

[0568] for emotion in emotions:

[0569] Search for products that correspond to each emotion

[0570] recommended_products = product_recommendation_module.recommend_products(products, emotion)

[0571] Display on smart glasses (pseudocode)

[0572] smart_glasses_display(recommended_products)

[0573] break

[0574] camera.release()

[0575] cv2.destroyAllWindows()

[0576]

[0577] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0578] Step 1:

[0579] The server acquires the camera stream, specifically, it uses OpenCV to capture video frames in real time from the in-store camera, where the input is the video camera stream and the output is individual video frames.

[0580] Step 2:

[0581] The server uses an emotion recognition engine to detect faces and analyze emotions in video frames. It receives video frames as input and runs a face detection algorithm to identify the location of faces. It then analyzes facial expressions to identify emotions such as happiness, sadness, surprise, etc. The output is the identified emotion.

[0582] Step 3:

[0583] The server sends the analyzed emotion data to a product suggestion module. The product suggestion module receives the emotion data as input and searches for a list of products corresponding to the emotion. For example, if the emotion of joy is identified, it selects "luxury sofa" from the list. The output is a list of products corresponding to the emotion.

[0584] Step 4:

[0585] The server sends the generated product list to the smart glasses. The input is the product list, and the output is the information to be displayed on the smart glasses' display. The smart glasses then provide the appropriate product information to the store staff.

[0586] Step 5:

[0587] The user (store staff) using the device selects products to recommend to customers based on the product suggestions displayed on the smart glasses display, making it possible to recommend the most suitable products to customers in a timely manner.

[0588] Step 6:

[0589] Users (customers) can review the products offered and get more information about the products they are interested in. This interaction is recorded and may be reflected in future recommendations.

[0590] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0591] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0592] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0593] [Second embodiment]

[0594] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0595] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0596] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0597] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0598] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0599] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0600] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0601] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0602] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0603] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0604] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0605] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0606] The present invention is a photo management AI system that efficiently organizes, manages, and searches large numbers of photos taken by users. Specific embodiments of each function are described below.

[0607] Auto-organization function

[0608] 1. When a device takes a photo, it collects its metadata (date, time, location, etc.).

[0609] 2. Photos are classified by event or season based on the metadata collected by the device. The classification algorithm analyzes timestamps based on the date and time of the photo and location based on GPS data.

[0610] 3. Your device will automatically create albums based on the classification results, for example, grouping photos taken during a specific period into a "Summer Vacation 2023" album.

[0611] Similar photo cleaning function

[0612] 1. The device scans the photos in the photo library and extracts features from each photo (e.g., facial recognition, hue histogram, etc.).

[0613] 2. The device calculates the similarity between the photos based on the extracted features, for example, using a face recognition algorithm.

[0614] 3. The server clusters photos with high similarity and presents deletion candidates to the user. Photos that the user deems unnecessary are added to the deletion list.

[0615] Conversational photo search feature

[0616] 1. A user types or speaks a search query in natural language, for example, "Show me photos of birthday parties in 2022."

[0617] 2. The device converts the voice input into text, which is then sent to the server.

[0618] 3. The server analyzes the search query and searches for matching photos. The server sets search criteria based on the metadata and extracted features.

[0619] 4. The server sends the discovered photos to the device, which displays them to the user.

[0620] Examples:

[0621] Scenario 1: A user returns from a trip and wants to organize their photos.

[0622] 1. The app automatically organizes 100 photos taken by the user on their smartphone while traveling.

[0623] 2. The device creates an album called "Summer Travel 2023" based on the metadata.

[0624] 3. The device scans the photo library, identifies 15 similar photos (the same scene taken from different angles), and presents them as candidates for deletion.

[0625] 4. The user confirms the deletion and the similar photos are deleted.

[0626] Scenario 2: A user wants to show a particular photo to a friend.

[0627] 1. The user says, "Show me pictures of the beach from last summer's vacation."

[0628] 2. The device converts the speech into text and sends it to the server.

[0629] 3. The server analyzes the query and searches for matching photos.

[0630] 4. The server finds the corresponding photo and sends it to the device.

[0631] 5. The device displays the search results to the user.

[0632] As described above, the system of the present invention significantly improves the efficiency of the user's photo management work, and enables the user to instantly find the photos they need.

[0633] The processing flow will be explained below.

[0634] Auto-organization process steps

[0635] Step 1:

[0636] The user takes a photo.

[0637] Use the camera function of your smartphone to capture a specific scene.

[0638] Step 2:

[0639] The device collects the photo's metadata.

[0640] Obtains the shooting date and time, shooting location (GPS information), camera settings (exposure, ISO, etc.), and scene information.

[0641] Step 3:

[0642] Your device will categorize your photos based on their metadata.

[0643] Analyze photos by event or season based on the date and time of the photo. For example, photos taken within a specific period of time can be classified as "summer events."

[0644] Step 4:

[0645] The device will automatically create albums with categorized photos.

[0646] Create an album called something like "Summer Trip 2023" and add relevant photos to it.

[0647] Similar Photo Cleanup Processing Steps

[0648] Step 1:

[0649] Your device will scan your photo library.

[0650] Load all the photo files and prepare them.

[0651] Step 2:

[0652] The device extracts features from each photo.

[0653] Features are calculated using algorithms such as face recognition, color histogram, and composition analysis.

[0654] Step 3:

[0655] The server calculates the similarity based on the extracted features.

[0656] The cosine similarity and Euclidean distance between feature vectors are calculated to evaluate which photos are similar.

[0657] Step 4:

[0658] The server clusters photos with high similarity.

[0659] Photos with high similarity are grouped into the same cluster.

[0660] Step 5:

[0661] The server presents the deletion candidate list to the user.

[0662] It displays a list of the results and allows the user to choose which photos to delete.

[0663] Step 6:

[0664] The server deletes unwanted photos based on the user's selection.

[0665] Delete unwanted photos from your photo library to free up storage space.

[0666] Conversational Photo Search Processing Steps

[0667] Step 1:

[0668] The user types or speaks a photo search query in natural language.

[0669] For example, type or say, "Show me pictures of Okinawa from last summer vacation."

[0670] Step 2:

[0671] The device converts the speech into text (in the case of voice input).

[0672] A speech recognition algorithm is used to convert instructions into text.

[0673] Step 3:

[0674] The device sends a text query to the server.

[0675] The textual query is sent to the server to make a search request.

[0676] Step 4:

[0677] The server parses the query.

[0678] It uses natural language processing models to interpret the intent of the query and set appropriate search criteria.

[0679] Step 5:

[0680] The server searches the photo library.

[0681] Find photos in your photo library based on the search criteria you set.

[0682] Step 6:

[0683] The server sends the search results to the terminal.

[0684] Sends a list of found photos to your device.

[0685] Step 7:

[0686] The terminal displays the search results to the user.

[0687] Display search results in a gallery view so users can browse photos.

[0688] The above is the specific processing flow for each function of the photo management AI system.

[0689] Example 1

[0690] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0691] In today's world, people take a huge number of photos using smartphones and digital cameras, but it is extremely difficult to efficiently organize these photos and quickly search for the ones they need. Furthermore, when there are a large number of similar photos, the task of reviewing and deleting them one by one takes time and effort. To solve these issues, a system is needed that can automatically organize and manage photos and easily delete similar photos.

[0692] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0693] In this invention, the server includes a means for collecting metadata of captured photos, a means for classifying photos by event or season based on the collected metadata, and a means for automatically creating albums of the classified photos. This allows for efficient organization of photos and quick search for specific photos. The server also includes a means for extracting photo features, a means for calculating similarity based on the extracted features, a means for clustering highly similar photos, a means for presenting photos that are candidates for deletion based on the clustering results, and a means for deleting unnecessary photos. This allows for efficient identification of similar photos and easy deletion of unnecessary photos.

[0694] The "user who took the photo" is a user who uses the camera function to acquire a still image.

[0695] A "metadata collection means" is a device or software that has the ability to automatically obtain additional information associated with a photograph, such as the date and time the photograph was taken, the location, and camera settings.

[0696] A "means for categorizing photos by event or season" is a device or software that has algorithms or functionality that analyzes collected metadata and groups photos based on specific times or events.

[0697] The "means for automatically creating an album of classified photographs" refers to a device or software that has the function of organizing photographs based on the classification results and arranging them into an album format that is easy for the user to view and manage.

[0698] A "means for extracting features" is a device or software that has the function of analyzing important visual features such as facial recognition and hue histograms in photographs and extracting them as numerical data.

[0699] The "means for calculating similarity" is a device or software that has the function of calculating and comparing the similarity between different photographs based on the extracted feature amounts.

[0700] The "clustering means" is a device or software that has an algorithm or function for grouping photos based on the calculated similarity and grouping photos that belong to the same group together.

[0701] The "means for presenting photos as candidates for deletion" refers to a device or software that has the function of displaying to the user photos that may be deemed unnecessary from among the highly similar photos identified by clustering, and suggesting that they be deleted.

[0702] A "means for entering search queries or providing voice commands using natural language" is a device or software that has an interface that allows a user to enter spoken commands to search for photos, or that has the ability to convert voice input into text.

[0703] A "means for searching for relevant photos within a photo library" is a device or software that has the function of identifying matching photos within a photo library using metadata and features based on an analyzed search query.

[0704] The "means for displaying search results to the user" refers to a device or software that has the function of displaying the searched photographs in a form that can be viewed by the user.

[0705] This invention relates to a photo management AI system for efficiently organizing, managing, and searching large numbers of photos taken by users. This system is mainly composed of a terminal and a server, and specific embodiments thereof are described in detail below.

[0706] Auto-organization function

[0707] When a device takes a photo, its metadata (such as the date and time the photo was taken and the location) is automatically collected. This collection is done using the device's camera and GPS functions. Based on the collected metadata, the device analyzes the timestamp and GPS data to classify the photos by event or season. Categorized photos are identified by a classification algorithm within the system, and the device automatically creates an album. For example, an album called "Summer Vacation 2023" is automatically generated.

[0708] Similar photo cleaning function

[0709] The device scans the photos in the photo library and extracts features such as facial recognition and hue histograms. Image processing software, for example, is used to extract the features. The device then calculates the similarity between photos based on the extracted features. A numerical data comparison algorithm is used to calculate this similarity. The server then clusters photos with high similarity and presents them as candidates for deletion. The server also manages the deletion of photos that the user deems unnecessary.

[0710] Conversational photo search feature

[0711] The user types or speaks a search query in natural language. Speech recognition software converts the query into text. The converted text is sent to a server, which analyzes the search query. The server sets search criteria based on metadata and extracted features and searches for matching photos in the photo library. The searched photos are sent from the server to the device, where they are displayed to the user.

[0712] Specific examples

[0713] For example, after a user returns from a trip, 100 photos taken on a smartphone are automatically organized. The device creates an album called "Summer 2023 Trip" based on the metadata. The device then scans the photo library and identifies 15 similar photos (the same scene taken from different angles) and presents them as candidates for deletion. The user can then confirm the deletion and delete the similar photos.

[0714] Or, if a user wants to show a friend a specific photo, they can say, "Show me photos from the beach last summer." The device converts the speech to text and sends the text to the server. The server analyzes the query, finds photos with relevant metadata, and sends those photos to the device. The device then displays the search results to the user.

[0715] This system significantly improves the efficiency of photo management and allows users to quickly find the photos they need. These specific actions allow users to organize and share their memories more easily, without the hassle of daily photo management.

[0716] Prompt Sentence Examples

[0717] 1. "Organize your summer 2022 travel photos"

[0718] 2. "Show me your Halloween pictures from October."

[0719] 3. "Add the latest family photo to my list for deletion."

[0720] 4. "Looking for photos from last year's Christmas party."

[0721] As described above, the present invention provides a system that supports the effective organization, management, and search of photos in a variety of situations.

[0722] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0723] Auto-organization function

[0724] Step 1:

[0725] When a device takes a photo, it automatically collects metadata such as the date, time and location of the photo.

[0726] Input: The photo taken

[0727] Output: Metadata added to photo files (timestamp, GPS information, etc.)

[0728] How it works: When a smartphone's camera takes a photo, the built-in GPS module and timestamp function retrieve the corresponding metadata and embed it in the photo file.

[0729] Step 2:

[0730] Based on the metadata collected by the device, timestamp analysis and GPS data analysis are performed to classify photos by event or season.

[0731] Input: Photo file with metadata

[0732] Output: Photo files with classification tags

[0733] What it does: Metadata analysis algorithms analyze timestamps and GPS data to add classification tags to photos, such as "Summer 2023 Travel."

[0734] Step 3:

[0735] Based on the classification results, the device will automatically create an album called "Summer Vacation 2023."

[0736] Input: Photo files with classification tags

[0737] Output: Auto-generated album folder

[0738] What it does: The photo management software creates folders based on classification tags and moves or copies relevant photos into those folders.

[0739] Similar photo cleaning function

[0740] Step 1:

[0741] The device scans photos in the photo library and extracts features such as facial recognition and hue histogram.

[0742] Input: All photos in your photo library

[0743] Output: Feature data for each photo

[0744] What it does: The facial recognition algorithm identifies faces in a photo and extracts their features as numerical data. In addition, it calculates a hue histogram and stores that as a feature.

[0745] Step 2:

[0746] The device calculates the similarity between photos based on the extracted features.

[0747] Input: Feature data

[0748] Output: Similarity score between each photo

[0749] What it does: It runs an algorithm that compares numerical data and calculates the similarity between photos using a score between 0 and 1.

[0750] Step 3:

[0751] The server clusters photos with high similarity and presents them as candidates for deletion.

[0752] Input: Similarity score

[0753] Output: List of photos to delete

[0754] What it does: Groups photos based on similarity scores and prompts the user, "Do you want to delete these photos? They're similar."

[0755] Step 4:

[0756] The user confirms the deletion and deletes the unwanted photos.

[0757] Input: List of photos to delete

[0758] Output: A cleaned-up photo library

[0759] Specific behavior: When the user presses the confirmation button, the photo data to be deleted will be erased from the system.

[0760] Conversational photo search feature

[0761] Step 1:

[0762] The user types or speaks a search query in natural language.

[0763] Input: Natural language search query or voice input

[0764] Output: The search query as text

[0765] What it does: Your smartphone's voice recognition software converts your speech into text in real time.

[0766] Step 2:

[0767] The device converts the voice input into text and sends the text to the server.

[0768] Input: Voice input

[0769] Output: The search query converted to text

[0770] Specific operation: The speech recognition API analyzes the speech, converts it into a string, and then sends it to the server.

[0771] Step 3:

[0772] The server analyzes the search query and searches for matching photos.

[0773] Input: Search query converted to text

[0774] Output: A list of photos from the search results

[0775] How it works: Natural language processing algorithms analyze your query and match it with metadata and features in your photo library to find matching photos.

[0776] Step 4:

[0777] The server sends the discovered photos to the terminal, which displays them to the user.

[0778] Input: Search result photo list

[0779] Output: The photo that is displayed to the user

[0780] Specific operation: The server sends the search result photo data to the terminal, and the terminal visually presents the photo to the user through an interface that displays the photo.

[0781] (Application example 1)

[0782] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0783] There is a demand for a method to efficiently organize, manage, and search large volumes of photos in physical stores and other locations. The goal is to solve these issues by providing an automatic organization function that classifies photos by event or season based on their metadata, a function to remove similar photos, and a function to search for photos using natural language or voice input. It is also important to efficiently analyze user input and provide fast search results.

[0784] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0785] In this invention, the server includes: a means for collecting users who have taken photos and metadata of the photos; a means for classifying photos by event or season based on the collected metadata; a means for automatically creating albums of the classified photos; a means for accepting user voice or text input; a means for converting the input voice into text; a means for analyzing the input using an AI model; a means for searching for relevant photos from the photo library based on the analysis results; and a means for displaying search results to the user. This allows users to easily organize and manage large numbers of photos and quickly find the photos they need. It also improves the accuracy of photo searches and reduces the burden on users.

[0786] "Photo metadata" is information related to a photograph that has been taken, and includes attribute information such as the date and time of the photograph and the location where the photograph was taken.

[0787] "Classifying by event or season" is the process of classifying photos according to specific events (e.g., birthdays, Christmas) or seasons (e.g., spring, summer) based on collected metadata.

[0788] "Automatically create albums" is a function that automatically generates albums according to specific themes or categories based on classified photos.

[0789] "Accepting user voice or text input" refers to a means for recognizing user voice instructions or text-entered search queries.

[0790] "Converting input speech to text" is the process of using speech recognition technology to convert user-entered voice instructions into text format.

[0791] An "AI model" refers to an artificial intelligence algorithm that has been trained to perform tasks such as natural language processing and image recognition.

[0792] "Searching for relevant photos in the photo library" refers to the process of searching for related photos from the photos in the library based on the analyzed data.

[0793] "Displaying search results to the user" refers to the function of displaying the searched photos on the user's device.

[0794] "Photo features" are numerical representations of specific patterns or structures within an image, and are information extracted in the form of facial recognition, hue histograms, etc.

[0795] "Calculating similarity" is a process of numerically evaluating the similarity between photos based on the extracted features.

[0796] "Clustering" refers to an algorithm that groups photos that are highly similar.

[0797] "Presenting photos as candidates for deletion" refers to the process of presenting photos that are deemed unnecessary based on the clustering results to the user and asking for confirmation of deletion.

[0798] "Converting proper nouns, proper names, personal names, and company names into generic names" is the process of replacing specific names or names with more abstract and generic expressions.

[0799] "Noun to superordinate representation" is the process of generalizing a detailed noun representation to its superordinate category.

[0800] "Generating prompt sentences" refers to the process of creating appropriate input sentences for a generative AI model in response to a user query or request.

[0801] The present invention is a system that allows users to efficiently organize, manage, and search for large amounts of photos in a physical store. Specific embodiments of each function are described below.

[0802] Auto-organization function

[0803] The server first collects metadata (such as the date and time of the photo taken, location, etc.) from the photos taken by the device. Based on this metadata, the photos are classified by event or season. Classification involves analyzing timestamps and GPS data. Albums are automatically created based on the classified photos. For example, photos taken in a particular season would be grouped together in a "2023 Summer Campaign" album.

[0804] Conversational photo search feature

[0805] Users can provide a photo search query via voice or text input. For example, a search request might be, "Show me photos of last year's Christmas decorations." The device converts the voice input into text, which is sent to the server. The server uses a generative AI model to analyze the query and search the photo library for matching photos. The search results are sent to the device and displayed to the user.

[0806] Similar photo cleaning function

[0807] The server scans the photos in the photo library and extracts the features of each photo (face recognition, hue histogram, etc.). Based on the extracted features, the similarity between photos is calculated. Photos with high similarity are clustered and presented to the user as candidates for deletion. Photos that the user deems unnecessary are added to the deletion list.

[0808] Explanation of program processing

[0809] The functions of this invention are implemented using smartphones, tablets, and computers as hardware, and the following libraries as software:

[0810] OpenCV: Used for image processing and face recognition.

[0811] scikit-learn: Used for similarity calculations and clustering.

[0812] transformers: Used for NLP processing and generative AI models.

[0813] Specific examples

[0814] Automatic organization: When a store runs a new spring product campaign, related photos are automatically organized into an album called "Spring 2023 New Product Campaign."

[0815] Conversational photo search: When a store manager says, "Show me pictures of last year's Halloween display," the system searches for and displays relevant photos.

[0816] Prompt Sentence Examples

[0817] Below is an example of a prompt sentence that is input to the NLP model.

[0818] "Show me a picture of last year's Christmas decorations."

[0819] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0820] Step 1:

[0821] The device takes a photo and collects its metadata (date, time, location, etc.). The collected metadata is sent to the server. The input is the photo and its metadata, and the output is the metadata sent to the server.

[0822] Step 2:

[0823] The server analyzes the collected metadata and classifies each photo by event or season. For example, photos taken during the summer are classified as a "summer album." The input is the metadata, and the output is the classification result.

[0824] Step 3:

[0825] Based on the classification results, the server automatically creates a photo album. For example, an album titled "2023 Summer Campaign" is generated. The input is the classification results, and the output is the generated album.

[0826] Step 4:

[0827] A user inputs a photo search query in natural language, either by voice or text, such as a request like "Show me pictures of last year's Christmas decorations." The input is the user query, and the output is speech or text data.

[0828] Step 5:

[0829] The device converts voice input into text. Using voice recognition technology, the input voice is converted into text. The converted text is sent to the server. The input is voice data, and the output is text data.

[0830] Step 6:

[0831] The server uses a generative AI model to analyze the input text query. The analysis identifies the conditions for understanding the query and identifying relevant photos. The input is text data, and the output is the analysis results.

[0832] Step 7:

[0833] The server searches for relevant photos in the photo library based on the analysis results. For example, a photo of "last year's Christmas decorations" is searched for based on metadata and features. The input is the analysis results, and the output is the relevant photos.

[0834] Step 8:

[0835] The server sends the search results to the device, which then displays them to the user. The user can view the photos displayed on the screen. The input is the search results, and the output is the displayed photos.

[0836] Step 9:

[0837] It scans photos in your photo library and extracts features for each photo (face recognition, hue histogram, etc.). The input is the photos in your photo library, and the output is the extracted features.

[0838] Step 10:

[0839] The server calculates the similarity between photos based on the extracted features. For example, it evaluates the similarity numerically using facial recognition or hue information. The input is the features, and the output is the similarity calculation result.

[0840] Step 11:

[0841] Photos with high similarity are clustered and presented to the user as candidates for deletion. Photos that the user deems unnecessary are added to a deletion list. The input is the similarity calculation result, and the output is a presentation of deletion candidates and a deletion list.

[0842] Step 12:

[0843] The user confirms the deletion and the unwanted photos are removed from the photo library. The input is the deletion list and the output is the updated photo library.

[0844] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0845] This invention combines a photo management AI system with an emotion engine that recognizes user emotions, allowing for efficient organization, management, and search of large numbers of photos taken by users, and providing interactions that respond to the user's emotions. Specific embodiments of each function are described below.

[0846] Auto-organization function

[0847] 1. When a device takes a photo, it collects its metadata (date, time, location, etc.).

[0848] 2. Photos are classified by event or season based on the metadata collected by the device. The classification algorithm analyzes timestamps based on the date and time of the photo and location based on GPS data.

[0849] 3. Your device will automatically create albums based on the classification results. For example, photos taken within a specific period will be grouped into a "Summer Vacation 2023" album.

[0850] Similar photo cleaning function

[0851] 1. The device scans the photos in the photo library and extracts features from each photo (e.g., facial recognition, hue histogram, etc.).

[0852] 2. The device calculates the similarity between the photos based on the extracted features, for example, using a face recognition algorithm.

[0853] 3. The server clusters photos with high similarity and presents deletion candidates to the user. Photos that the user deems unnecessary are added to the deletion list.

[0854] Conversational photo search feature

[0855] 1. A user types or speaks a search query in natural language, for example, "Show me photos of birthday parties in 2022."

[0856] 2. The device converts the voice input into text, which is then sent to the server.

[0857] 3. The server analyzes the search query and searches for matching photos. The server sets search criteria based on the metadata and extracted features.

[0858] 4. The server sends the discovered photos to the device, which displays them to the user.

[0859] Emotion recognition and linking functions using an emotion engine

[0860] 1. The device captures the facial expressions and voice of the user viewing the photo in real time and sends them to the emotion engine.

[0861] 2. The emotion engine analyzes the user's emotional state and identifies emotions such as joy, sadness, surprise, etc.

[0862] 3. The device optimizes the display order of photos according to the user's emotions. For example, if the user is happy, photos of happy memories will be displayed first.

[0863] 4. The emotion engine automatically generates albums based on specific emotional states and selects photos that fit the user's mood. For example, if the user is sad, it will suggest an album of "fun trips."

[0864] Examples:

[0865] Scenario 1: A user returns from a trip and wants to organize their photos.

[0866] 1. The app automatically organizes 100 photos taken by the user on their smartphone while traveling.

[0867] 2. The device creates an album called "Summer Travel 2023" based on the metadata.

[0868] 3. The device scans the photo library, identifies 15 similar photos (the same scene taken from different angles), and presents them as candidates for deletion.

[0869] 4. The user confirms the deletion and the similar photos are deleted.

[0870] Scenario 2: A user wants to show a particular photo to a friend.

[0871] 1. The user says, "Show me pictures of the beach from last summer's vacation."

[0872] 2. The device converts the speech into text and sends it to the server.

[0873] 3. The server analyzes the query and searches for matching photos.

[0874] 4. The server finds the corresponding photo and sends it to the device.

[0875] 5. The device displays the search results to the user.

[0876] Scenario 3: When the user is in a particular emotional state.

[0877] 1. The user smiles while viewing the photo.

[0878] 2. The device sends the user's facial expressions to the emotion engine.

[0879] 3. The emotion engine recognizes the user's emotion as "joy."

[0880] 4. The device will prioritize displaying photos of happy events according to the emotion of "joy."

[0881] 5. The emotion engine automatically generates an album of "happy memories" and suggests them to the user.

[0882] As described above, the system of the present invention significantly improves the efficiency of users' photo management tasks, and by providing interactions that respond to the user's emotions, it realizes a more personalized photo organization, management, and search experience.

[0883] The processing flow will be explained below.

[0884] Auto-organization process steps

[0885] Step 1:

[0886] The user takes a photo.

[0887] Use the camera function of your smartphone to capture a specific scene.

[0888] Step 2:

[0889] The device collects the photo's metadata.

[0890] Obtains the shooting date and time, shooting location (GPS information), camera settings (exposure, ISO, etc.), and scene information.

[0891] Step 3:

[0892] Your device will categorize your photos based on their metadata.

[0893] Analyze photos by event or season based on the date and time of the photo. For example, photos taken within a specific period of time can be classified as "summer events."

[0894] Step 4:

[0895] The device will automatically create albums with categorized photos.

[0896] Create an album called something like "Summer Trip 2023" and add relevant photos to it.

[0897] Similar Photo Cleanup Processing Steps

[0898] Step 1:

[0899] Your device will scan your photo library.

[0900] Load all the photo files and prepare them.

[0901] Step 2:

[0902] The device extracts features from each photo.

[0903] Features are calculated using algorithms such as face recognition, color histogram, and composition analysis.

[0904] Step 3:

[0905] The server calculates the similarity based on the extracted features.

[0906] The cosine similarity and Euclidean distance between feature vectors are calculated to evaluate which photos are similar.

[0907] Step 4:

[0908] The server clusters photos with high similarity.

[0909] Photos with high similarity are grouped into the same cluster.

[0910] Step 5:

[0911] The server presents the deletion candidate list to the user.

[0912] It displays a list of the results and allows the user to choose which photos to delete.

[0913] Step 6:

[0914] The server deletes unwanted photos based on the user's selection.

[0915] Delete unwanted photos from your photo library to free up storage space.

[0916] Conversational Photo Search Processing Steps

[0917] Step 1:

[0918] The user types or speaks a photo search query in natural language.

[0919] For example, type or say, "Show me pictures of Okinawa from last summer vacation."

[0920] Step 2:

[0921] The device converts the speech into text (in the case of voice input).

[0922] A speech recognition algorithm is used to convert instructions into text.

[0923] Step 3:

[0924] The device sends a text query to the server.

[0925] The textual query is sent to the server to make a search request.

[0926] Step 4:

[0927] The server parses the query.

[0928] It uses natural language processing models to interpret the intent of the query and set appropriate search criteria.

[0929] Step 5:

[0930] The server searches the photo library.

[0931] Find photos in your photo library based on the search criteria you set.

[0932] Step 6:

[0933] The server sends the search results to the terminal.

[0934] Sends a list of found photos to your device.

[0935] Step 7:

[0936] The terminal displays the search results to the user.

[0937] Display search results in a gallery view so users can browse photos.

[0938] Processing steps for emotion recognition and linkage functions by the emotion engine

[0939] Step 1:

[0940] The device captures the facial expressions and voice of the user viewing the photos in real time.

[0941] A camera and microphone are used to collect facial expression and voice data of the user.

[0942] Step 2:

[0943] The device sends facial expression data and voice data to the emotion engine.

[0944] The collected data is sent to an emotion engine to analyze the user's emotional state.

[0945] Step 3:

[0946] An emotion engine analyzes the user's emotional state.

[0947] It recognizes the user's emotions by applying algorithms that identify emotions such as joy, sadness, and surprise.

[0948] Step 4:

[0949] The device optimizes the display order of photos according to the user's emotions.

[0950] For example, if the user is happy, photos of happy memories are displayed preferentially.

[0951] Step 5:

[0952] The emotional engine automatically generates albums that correspond to specific emotional states.

[0953] Select photos that match the user's mood and create albums for "happy memories" or "refreshment."

[0954] Step 6:

[0955] The terminal proposes automatically generated albums to the user.

[0956] The app displays an album of "fun trips" and makes suggestions to the user.

[0957] The above is the specific processing flow for each function of the photo management AI system that combines the emotion engine.

[0958] Example 2

[0959] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0960] Conventional photo management systems have the drawback of making it difficult for users to effectively organize, manage, and search large numbers of photos, which is time-consuming. In addition, the photo viewing experience is not personalized based on the user's emotions, which hinders user satisfaction.

[0961] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0962] In this invention, the server includes means for collecting photo metadata, means for classifying photos by event or season, means for automatically creating albums of the classified photos, means for analyzing emotions, means for optimizing the display order of photos based on the emotion analysis results, and means for automatically creating albums according to specific themes based on the emotion analysis. This not only enables efficient organization and management of photos, but also provides a personalized photo viewing experience according to the user's emotions.

[0963] "User" refers to the person taking the photograph and using the system.

[0964] "Metadata" is information related to a photo, including details such as the date and time the photo was taken, the location, and camera settings.

[0965] "Classification" refers to the act of organizing photos into specific events or seasons based on collected metadata.

[0966] An "album" is a group of classified photos, which is automatically created based on a specific theme or period.

[0967] "Emotion analysis means" refers to technology for analyzing and identifying a user's emotional state from data such as facial expressions and voice.

[0968] "Optimizing the display order" refers to the act of adjusting the order of displayed photos based on the results of user sentiment analysis.

[0969] A "themed album" refers to an album that is automatically generated by collecting photos that correspond to a specific emotion or theme based on the results of emotional analysis.

[0970] "Features" are numerically quantified information about the content of a photograph, and refer to data that represents the characteristics of a photograph, such as facial recognition and hue histograms.

[0971] "Similarity" refers to a numerical measure for assessing the content similarity between photographs.

[0972] "Clustering" refers to a technique for grouping similar photos based on extracted features.

[0973] "Candidates for deletion" refers to photos that the user has determined to be unnecessary or that are highly similar and therefore useless.

[0974] "Search query" refers to input instructions, specified by a user in natural language or speech, for searching for a particular photo.

[0975] "Parsing" refers to the process of breaking down the input search query and converting it into appropriate search terms.

[0976] This invention is a photo management AI system that enables efficient organization, management, and search of photos taken by users, and also provides interaction based on the user's emotions. This system collects and analyzes photo metadata, automatically classifies and creates albums, identifies similar photos and suggests deletion options, and optimizes photo display through emotion analysis.

[0977] When a user takes a photo on a device, metadata such as the date and time of the photo and the location are collected. The device then uses the camera API to obtain Exif information and collect the metadata. Based on this metadata, the photos are classified by specific events or seasons. Classification is performed by analyzing the timestamp of the photo and GPS data. This allows an album called, for example, "Summer Trip 2023" to be automatically created.

[0978] Next, the device scans the photos in the photo library and extracts features from each photo. Based on the extracted features, the similarity between photos is calculated. Photos with high similarity are grouped using a clustering algorithm and presented as candidates for deletion. For example, photos of the same scene taken from different angles may be grouped into a specific list.

[0979] With the conversational photo search feature, users enter search queries by voice, and the device converts the speech into text and sends it to the server. The server analyzes the search query, searches for matching photos in the photo library, and displays the results to the user. For example, a user can search for photos using a search query such as "Show me photos of the beach from last summer vacation."

[0980] With the emotion analysis function, the device captures the user's facial expressions and voice in real time and sends them to the emotion engine. The emotion engine uses facial expression recognition algorithms and voice emotion analysis algorithms to analyze the user's emotional state and identify emotions such as joy, sadness, and surprise. This allows the device to optimize the display order of photos according to the user's emotions. For example, if the user smiles while viewing photos, the emotion engine will recognize this as "joy" and prioritize displaying photos of happy events. In addition, the device can automatically generate an album of "happy memories" based on the emotion analysis and suggest them to the user.

[0981] The hardware used includes devices such as smartphones, tablets, and PCs, and the required software includes a camera API, a voice recognition API (such as Google Speech-to-Text API), and an emotion analysis algorithm (such as Face++ or EmoVu). Additionally, server-side search systems such as Elasticsearch and Lucene can be used.

[0982] With the above configuration, this system can significantly improve the efficiency of users' photo management tasks and provide a personalized photo browsing experience that reflects their emotions. Specific examples of prompts are as follows:

[0983] "Describe an automated photo organization system based on a generative AI model."

[0984] "Please explain with some examples how to manage photos using the emotion engine."

[0985] This allows users to easily search for specific photos and enjoy photos that match their emotions.

[0986] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0987] Step 1:

[0988] Taking photos and collecting metadata

[0989] When a user takes a photo, the device collects metadata in Exif format (such as the date and time the photo was taken, the location, and the camera settings).

[0990] Input: Photo data, metadata at the time of shooting

[0991] Output: A data object containing the photo and its metadata.

[0992] Specific operation: When the smartphone's camera application saves a photo, it simultaneously obtains the Exif information associated with the photo and stores it in an internal database.

[0993] Step 2:

[0994] Photo classification based on metadata

[0995] The device analyzes the metadata it collects and categorizes photos by event or season based on the date, time, and location they were taken.

[0996] Input: Photo data including metadata

[0997] Output: A list of classified photo data

[0998] What it does: Your device analyzes the timestamps and runs an algorithm to categorize the photos based on events, such as "Summer 2023" or "a specific trip."

[0999] Step 3:

[1000] Auto-create albums

[1001] The device automatically creates an album based on the classification results.

[1002] Input: A list of classified photo data

[1003] Output: Auto-generated album

[1004] Specific operation: The device creates an album named, for example, "Summer Travel 2023," and adds the corresponding photo data to the album.

[1005] Step 4:

[1006] Photo library scanning and feature extraction

[1007] The device scans photos in the photo library and extracts features such as facial recognition and hue histogram.

[1008] Input: Photo data in the photo library

[1009] Output: Feature data for each photo

[1010] Specific operation: The device uses an image processing library such as OpenCV to extract facial features and color distribution from each photo.

[1011] Step 5:

[1012] Calculating similarity between photos

[1013] The device calculates the similarity between photos based on the features extracted.

[1014] Input: Feature data

[1015] Output: Similarity scores and photo pairs

[1016] Specific operation: The device performs a similarity calculation method (e.g., Euclidean distance) based on face recognition and hue distribution to calculate a similarity score.

[1017] Step 6:

[1018] Clustering and suggesting deletion candidates

[1019] The server clusters photos with high similarity and presents them to the user as candidates for deletion.

[1020] Input: Similarity scores and photo pairs

[1021] Output: Clustering results and removal candidate list

[1022] What it does: The server uses a clustering algorithm such as DBSCAN to group similar photos and displays deletion candidates to the user on the interface.

[1023] Step 7:

[1024] Confirm and execute the deletion

[1025] The user selects unwanted photos from the presented list of photos to delete and confirms the deletion.

[1026] Input: Deletion candidate list and user selection

[1027] Output: Updated photo library

[1028] What happens: When the user presses the Delete button, the selected photo is deleted from the photo library.

[1029] Step 8:

[1030] Entering search queries in natural language

[1031] The user enters a search query by voice.

[1032] Input: Audio data

[1033] Output: Text query

[1034] Specific action: Say to your voice assistant or smart speaker, "Show me photos of the beach from last summer vacation."

[1035] Step 9:

[1036] Speech-to-text

[1037] The device converts the voice input into text and sends it to the server.

[1038] Input: Audio data

[1039] Output: Text data

[1040] What happens: A speech recognition API (e.g., Google Speech-to-Text API) analyzes the audio data and generates corresponding text.

[1041] Step 10:

[1042] Search query analysis and photo search

[1043] The server analyzes the text query and searches for matching photos in the photo library.

[1044] Input: Text query

[1045] Output: List of photo data

[1046] Specific operation: Analyzes queries using a natural language processing model (generative AI model) and searches for photos that match the criteria based on metadata and features.

[1047] Step 11:

[1048] Displaying search results

[1049] The server sends the search results to the terminal, which displays them to the user.

[1050] Input: List of photo data

[1051] Output: Displaying the photo on a user interface

[1052] Specific operation: The retrieved photos are displayed on the device display in album or thumbnail format.

[1053] Step 12:

[1054] Acquiring and sending emotion data

[1055] The device captures the user's facial expressions and voice in real time and sends them to the emotion engine.

[1056] Input: Real-time facial expression data, voice data

[1057] Output: Data for sentiment analysis

[1058] Specific behavior: Uses a camera and microphone to capture the user's facial expressions and voice in real time and send them to the emotion engine.

[1059] Step 13:

[1060] Sentiment Analysis and Identification

[1061] The emotion engine analyzes the user's emotional state and identifies emotions such as joy, sadness, surprise, etc.

[1062] Input: Facial expression data, voice data

[1063] Output: Emotional state label (e.g., happy, sad)

[1064] Specific operation: Identify emotions using facial expression recognition algorithms and voice emotion analysis algorithms.

[1065] Step 14:

[1066] Optimize the display order of photos

[1067] The device optimizes the display order of photos according to the user's emotions.

[1068] Input: Emotional state labels, photo data from the photo library

[1069] Output: A list of photos in an optimized display order

[1070] What it does: Based on your emotional state, it rearranges the photos in your photo library and prioritizes photos that evoke positive emotions.

[1071] Step 15:

[1072] Emotion-based album generation

[1073] The emotion engine automatically generates albums that correspond to specific emotional states and suggests them to the user.

[1074] Input: Emotional state labels, classified photo data

[1075] Output: Thematic albums

[1076] Specific operation: For example, if the user is sad, an album of photos of "happy memories" is automatically generated and displayed to the user.

[1077] (Application example 2)

[1078] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1079] In physical stores, there is a need to make optimal product recommendations in real time based on customer emotions. However, conventional methods have the problem that it is difficult for store staff to directly observe customers' emotions and recommend appropriate products, which takes time and effort. Furthermore, because it is not possible to accurately grasp customers' emotions, the recommended products often do not meet the customer's needs. To solve these problems, a system is needed that can recognize customers' emotions in real time and recommend products based on them.

[1080] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing customer emotions in real time, means for optimizing photos to be displayed based on the emotion recognition, means for providing interactions based on the emotion recognition, and means for suggesting target information based on the emotion recognition. This makes it possible to accurately grasp customer emotions and suggest optimal products based on that.

[1081] The "user who took the photo" refers to the person who captured the still image using a camera.

[1082] "Means for collecting metadata from photographs taken" refers to methods for automatically acquiring and storing relevant information such as the date and time a photograph was taken, the location where it was taken, and camera settings.

[1083] "Means for categorizing photos by event or season based on collected metadata" refers to a method for using acquired metadata information to group photos based on a specific time period or specific location.

[1084] "Means for automatically creating albums of classified photos" refers to a method for aggregating classified photos and organizing them into a collection based on a specific theme or event.

[1085] "Means for recognizing a user's emotions in real time" refers to technology that analyzes biometric information such as a user's facial expressions and voice, and identifies a specific emotional state in real time.

[1086] "Means for optimizing photos to be displayed based on emotion recognition" refers to technology that automatically adjusts the order and content of photos to be displayed according to the user's recognized emotion.

[1087] "Means for providing interactions based on emotion recognition" refers to technology that enables a system to automatically provide appropriate responses and actions in accordance with the user's emotional state.

[1088] "Means for suggesting targeted information based on emotion recognition" refers to technology that analyzes a user's emotional state and, based on this, recommends information on optimal products and services to the user.

[1089] This invention relates to a system that efficiently organizes and manages photos taken by users and suggests products based on the user's emotions. This system is particularly effective in brick-and-mortar stores, where store staff can use smart glasses to grasp customers' emotions in real time and make optimal product suggestions.

[1090] System configuration

[1091] The system of the present invention consists of the following major components:

[1092] 1. Smart Glasses

[1093] The smart glasses are equipped with a camera that captures the faces of customers in the store in real time, and also have a display that provides store staff with the necessary information instantly.

[1094] 2. Server

[1095] The server works in conjunction with an emotion recognition engine to analyze customer emotions and generate a list of product suggestions based on the analysis results.

[1096] 3. Emotion Recognition Engine

[1097] It is a software module that analyzes a customer's facial expressions and voice to identify their emotional state, such as joy, sadness, or surprise.

[1098] 4. Product Proposal Module

[1099] It includes an algorithm to suggest optimal products based on the analysis results of the emotion recognition engine.

[1100] Hardware and software used

[1101] Hardware:

[1102] Smart glasses (e.g. Google Glass, Vuzix Blade)

[1103] In-store camera

[1104] software:

[1105] OpenCV: Capture frames from camera stream

[1106] emotion_recognition_module: Detects customer faces from frames and analyzes emotions

[1107] product_recommendation_module: Search for the best products based on analyzed sentiment data

[1108] How it works

[1109] The server acquires the camera stream using OpenCV and captures frames. From these frames, the emotion recognition engine (emotion_recognition_module) analyzes the customer's facial expressions and identifies their emotional state. Based on the identified emotions, the product recommendation module (product_recommendation_module) generates an appropriate product list. The generated list is displayed on the smart glasses display, and store staff receive instructions on how to recommend the most suitable products to the customer.

[1110] Specific examples

[1111] A specific example of use in a physical store is shown below. For example, consider a case where store staff wear smart glasses to make product recommendations. When a customer enters the display area, the camera installed in the smart glasses captures the person's face and recognizes their emotions in real time. For example, if the customer smiles, the glasses will recognize that emotion as "happiness" and suggest a product that matches that emotion (e.g., a luxury sofa).

[1112] Prompt Sentence Examples

[1113] Below is an example of an input prompt for a generative AI model.

[1114] Recognize faces and estimate emotions. Recommend products based on specific emotions.

[1115] import cv2

[1116] import emotion_recognition_module

[1117] import product_recommendation_module

[1118] Get Camera Stream

[1119] camera = cv2.VideoCapture(0)

[1120] Product information list

[1121] products = [

[1122] {"name": "luxury sofa", "emotion": "joy"},

[1123] {"name": "Relax chair", "emotion": "sadness"},

[1124] {"name": "Energetic Lamp", "emotion": "surprise"}

[1125] ]

[1126] while True:

[1127] Capture frames from the camera stream

[1128] ret, frame = camera.read()

[1129] if not ret:

[1130] break

[1131] Detect faces from frames and analyze emotions

[1132] emotions = emotion_recognition_module.detect_emotion(frame)

[1133] for emotion in emotions:

[1134] Search for products that correspond to each emotion

[1135] recommended_products = product_recommendation_module.recommend_products(products, emotion)

[1136] Display on smart glasses (pseudocode)

[1137] smart_glasses_display(recommended_products)

[1138] break

[1139] camera.release()

[1140] cv2.destroyAllWindows()

[1141]

[1142] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1143] Step 1:

[1144] The server acquires the camera stream, specifically, it uses OpenCV to capture video frames in real time from the in-store camera, where the input is the video camera stream and the output is individual video frames.

[1145] Step 2:

[1146] The server uses an emotion recognition engine to detect faces and analyze emotions in video frames. It receives video frames as input and runs a face detection algorithm to identify the location of faces. It then analyzes facial expressions to identify emotions such as happiness, sadness, surprise, etc. The output is the identified emotion.

[1147] Step 3:

[1148] The server sends the analyzed emotion data to a product suggestion module. The product suggestion module receives the emotion data as input and searches for a list of products corresponding to the emotion. For example, if the emotion of joy is identified, it selects "luxury sofa" from the list. The output is a list of products corresponding to the emotion.

[1149] Step 4:

[1150] The server sends the generated product list to the smart glasses. The input is the product list, and the output is the information to be displayed on the smart glasses' display. The smart glasses then provide the appropriate product information to the store staff.

[1151] Step 5:

[1152] The user (store staff) using the device selects products to recommend to customers based on the product suggestions displayed on the smart glasses display, making it possible to recommend the most suitable products to customers in a timely manner.

[1153] Step 6:

[1154] Users (customers) can review the products offered and get more information about the products they are interested in. This interaction is recorded and may be reflected in future recommendations.

[1155] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1156] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1157] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1158] [Third embodiment]

[1159] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1160] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1161] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1162] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1163] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1164] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1165] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1166] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1167] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1168] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1169] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1170] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1171] The present invention is a photo management AI system that efficiently organizes, manages, and searches large numbers of photos taken by users. Specific embodiments of each function are described below.

[1172] Auto-organization function

[1173] 1. When a device takes a photo, it collects its metadata (date, time, location, etc.).

[1174] 2. Based on the metadata collected by the device, photos are classified by event or season. The classification algorithm analyzes timestamps based on the date and time of the photo and location based on GPS data.

[1175] 3. Your device will automatically create albums based on the classification results, for example, grouping photos taken during a specific period into a "Summer Vacation 2023" album.

[1176] Similar photo cleaning function

[1177] 1. The device scans the photos in the photo library and extracts features from each photo (e.g., facial recognition, hue histogram, etc.).

[1178] 2. The device calculates the similarity between the photos based on the extracted features, for example, using a face recognition algorithm.

[1179] 3. The server clusters photos with high similarity and presents deletion candidates to the user. Photos that the user deems unnecessary are added to the deletion list.

[1180] Conversational photo search feature

[1181] 1. A user types or speaks a search query in natural language, for example, "Show me photos from birthday parties in 2022."

[1182] 2. The device converts the voice input into text, which is then sent to the server.

[1183] 3. The server analyzes the search query and searches for matching photos. The server sets search criteria based on the metadata and extracted features.

[1184] 4. The server sends the discovered photos to the device, which displays them to the user.

[1185] Examples:

[1186] Scenario 1: A user returns from a trip and wants to organize their photos.

[1187] 1. The app automatically organizes 100 photos taken by the user on their smartphone while traveling.

[1188] 2. The device creates an album called "Summer Travel 2023" based on the metadata.

[1189] 3. The device scans the photo library, identifies 15 similar photos (the same scene taken from different angles), and presents them as candidates for deletion.

[1190] 4. The user confirms the deletion and the similar photos are deleted.

[1191] Scenario 2: A user wants to show a particular photo to a friend.

[1192] 1. The user says, "Show me pictures of the beach from last summer's vacation."

[1193] 2. The device converts the speech into text and sends it to the server.

[1194] 3. The server analyzes the query and searches for matching photos.

[1195] 4. The server finds the corresponding photo and sends it to the device.

[1196] 5. The device displays the search results to the user.

[1197] As described above, the system of the present invention significantly improves the efficiency of the user's photo management work, and enables the user to instantly find the photos they need.

[1198] The processing flow will be explained below.

[1199] Auto-organization process steps

[1200] Step 1:

[1201] The user takes a photo.

[1202] Use the camera function of your smartphone to capture a specific scene.

[1203] Step 2:

[1204] The device collects the photo's metadata.

[1205] Obtains the shooting date and time, shooting location (GPS information), camera settings (exposure, ISO, etc.), and scene information.

[1206] Step 3:

[1207] Your device will categorize your photos based on their metadata.

[1208] Analyze photos by event or season based on the date and time of the photo. For example, photos taken within a specific period of time can be classified as "summer events."

[1209] Step 4:

[1210] The device will automatically create albums with categorized photos.

[1211] Create an album called something like "Summer Trip 2023" and add relevant photos to it.

[1212] Similar Photo Cleanup Processing Steps

[1213] Step 1:

[1214] Your device will scan your photo library.

[1215] Load all the photo files and prepare them.

[1216] Step 2:

[1217] The device extracts features from each photo.

[1218] Features are calculated using algorithms such as face recognition, color histogram, and composition analysis.

[1219] Step 3:

[1220] The server calculates the similarity based on the extracted features.

[1221] The cosine similarity and Euclidean distance between feature vectors are calculated to evaluate which photos are similar.

[1222] Step 4:

[1223] The server clusters photos with high similarity.

[1224] Photos with high similarity are grouped into the same cluster.

[1225] Step 5:

[1226] The server presents the deletion candidate list to the user.

[1227] It displays a list of the results and allows the user to choose which photos to delete.

[1228] Step 6:

[1229] The server deletes unwanted photos based on the user's selection.

[1230] Delete unwanted photos from your photo library to free up storage space.

[1231] Conversational Photo Search Processing Steps

[1232] Step 1:

[1233] The user types or speaks a photo search query in natural language.

[1234] For example, type or say, "Show me pictures of Okinawa from last summer vacation."

[1235] Step 2:

[1236] The device converts the speech into text (in the case of voice input).

[1237] A speech recognition algorithm is used to convert instructions into text.

[1238] Step 3:

[1239] The device sends a text query to the server.

[1240] The textual query is sent to the server to make a search request.

[1241] Step 4:

[1242] The server parses the query.

[1243] It uses natural language processing models to interpret the intent of the query and set appropriate search criteria.

[1244] Step 5:

[1245] The server searches the photo library.

[1246] Find photos in your photo library based on the search criteria you set.

[1247] Step 6:

[1248] The server sends the search results to the terminal.

[1249] Sends a list of found photos to your device.

[1250] Step 7:

[1251] The terminal displays the search results to the user.

[1252] Display search results in a gallery view so users can browse photos.

[1253] The above is the specific processing flow for each function of the photo management AI system.

[1254] Example 1

[1255] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1256] In today's world, people take a huge number of photos using smartphones and digital cameras, but it is extremely difficult to efficiently organize these photos and quickly search for the ones they need. Furthermore, when there are a large number of similar photos, the task of reviewing and deleting them one by one takes time and effort. To solve these issues, a system is needed that can automatically organize and manage photos and easily delete similar photos.

[1257] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1258] In this invention, the server includes a means for collecting metadata of captured photos, a means for classifying photos by event or season based on the collected metadata, and a means for automatically creating albums of the classified photos. This allows for efficient organization of photos and quick search for specific photos. The server also includes a means for extracting photo features, a means for calculating similarity based on the extracted features, a means for clustering highly similar photos, a means for presenting photos that are candidates for deletion based on the clustering results, and a means for deleting unnecessary photos. This allows for efficient identification of similar photos and easy deletion of unnecessary photos.

[1259] The "user who took the photo" is a user who uses the camera function to acquire a still image.

[1260] A "metadata collection means" is a device or software that has the ability to automatically obtain additional information associated with a photograph, such as the date and time the photograph was taken, the location, and the camera settings.

[1261] A "means for categorizing photos by event or season" is a device or software that has algorithms or functionality that analyzes collected metadata and groups photos based on specific times or events.

[1262] The "means for automatically creating an album of classified photographs" refers to a device or software that has the function of organizing photographs based on the classification results and arranging them into an album format that is easy for the user to view and manage.

[1263] A "means for extracting features" is a device or software that has the function of analyzing important visual features such as facial recognition and hue histograms in photographs and extracting them as numerical data.

[1264] The "means for calculating similarity" is a device or software that has the function of calculating and comparing the similarity between different photographs based on the extracted feature amounts.

[1265] The "clustering means" is a device or software that has an algorithm or function for grouping photos based on the calculated similarity and grouping photos that belong to the same group together.

[1266] The "means for presenting photos as candidates for deletion" refers to a device or software that has the function of displaying to the user photos that may be deemed unnecessary from among the highly similar photos identified by clustering, and suggesting that they be deleted.

[1267] A "means for entering search queries or providing voice commands using natural language" is a device or software that has an interface that allows a user to enter spoken commands to search for photos, or that has the ability to convert voice input into text.

[1268] A "means for searching for relevant photos within a photo library" is a device or software that has the function of identifying matching photos within a photo library using metadata and features based on an analyzed search query.

[1269] The "means for displaying search results to the user" refers to a device or software that has the function of displaying the searched photographs in a form that can be viewed by the user.

[1270] This invention relates to a photo management AI system for efficiently organizing, managing, and searching large numbers of photos taken by users. This system is mainly composed of a terminal and a server, and specific embodiments thereof are described in detail below.

[1271] Auto-organization function

[1272] When a device takes a photo, its metadata (such as the date and time the photo was taken and the location) is automatically collected. This collection is done using the device's camera and GPS functions. Based on the collected metadata, the device analyzes the timestamp and GPS data to classify the photos by event or season. Categorized photos are identified by a classification algorithm within the system, and the device automatically creates an album. For example, an album called "Summer Vacation 2023" is automatically generated.

[1273] Similar photo cleaning function

[1274] The device scans the photos in the photo library and extracts features such as facial recognition and hue histograms. Image processing software, for example, is used to extract the features. The device then calculates the similarity between photos based on the extracted features. A numerical data comparison algorithm is used to calculate this similarity. The server then clusters photos with high similarity and presents them as candidates for deletion. The server also manages the deletion of photos that the user deems unnecessary.

[1275] Conversational photo search feature

[1276] The user types or speaks a search query in natural language. Speech recognition software converts the query into text. The converted text is sent to a server, which analyzes the search query. The server sets search criteria based on metadata and extracted features and searches for matching photos in the photo library. The searched photos are sent from the server to the device, where they are displayed to the user.

[1277] Specific examples

[1278] For example, after a user returns from a trip, 100 photos taken on a smartphone are automatically organized. The device creates an album called "Summer 2023 Trip" based on the metadata. The device then scans the photo library and identifies 15 similar photos (the same scene taken from different angles) and presents them as candidates for deletion. The user can then confirm the deletion and delete the similar photos.

[1279] Or, if a user wants to show a friend a specific photo, they can say, "Show me photos from the beach last summer." The device converts the speech to text and sends the text to the server. The server analyzes the query, finds photos with relevant metadata, and sends those photos to the device. The device then displays the search results to the user.

[1280] This system significantly improves the efficiency of photo management and allows users to quickly find the photos they need. These specific actions allow users to organize and share their memories more easily, without the hassle of daily photo management.

[1281] Prompt Sentence Examples

[1282] 1. "Organize your summer 2022 travel photos"

[1283] 2. "Show me your Halloween pictures from October."

[1284] 3. "Add the latest family photo to my list for deletion."

[1285] 4. "Looking for photos from last year's Christmas party."

[1286] As described above, the present invention provides a system that supports the effective organization, management, and search of photos in a variety of situations.

[1287] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1288] Auto-organization function

[1289] Step 1:

[1290] When a device takes a photo, it automatically collects metadata such as the date, time and location of the photo.

[1291] Input: The photo taken

[1292] Output: Metadata added to photo files (timestamp, GPS information, etc.)

[1293] How it works: When a smartphone's camera takes a photo, the built-in GPS module and timestamp function retrieve the corresponding metadata and embed it in the photo file.

[1294] Step 2:

[1295] Based on the metadata collected by the device, timestamp analysis and GPS data analysis are performed to classify photos by event or season.

[1296] Input: Photo file with metadata

[1297] Output: Photo files with classification tags

[1298] What it does: Metadata analysis algorithms analyze timestamps and GPS data to add classification tags to photos, such as "Summer 2023 Travel."

[1299] Step 3:

[1300] Based on the classification results, the device will automatically create an album called "Summer Vacation 2023."

[1301] Input: Photo files with classification tags

[1302] Output: Auto-generated album folder

[1303] What it does: The photo management software creates folders based on classification tags and moves or copies relevant photos into those folders.

[1304] Similar photo cleaning function

[1305] Step 1:

[1306] The device scans the photos in your photo library and extracts features such as facial recognition and hue histogram.

[1307] Input: All photos in your photo library

[1308] Output: Feature data for each photo

[1309] What it does: The facial recognition algorithm identifies faces in a photo and extracts their features as numerical data. In addition, it calculates a hue histogram and stores that as a feature.

[1310] Step 2:

[1311] The device calculates the similarity between photos based on the extracted features.

[1312] Input: Feature data

[1313] Output: Similarity score between each photo

[1314] What it does: It runs an algorithm that compares numerical data and calculates the similarity between photos using a score between 0 and 1.

[1315] Step 3:

[1316] The server clusters photos with high similarity and presents them as candidates for deletion.

[1317] Input: Similarity score

[1318] Output: List of photos to delete

[1319] What it does: Groups photos based on similarity scores and prompts the user, "Do you want to delete these photos? They're similar."

[1320] Step 4:

[1321] The user confirms the deletion and deletes the unwanted photos.

[1322] Input: List of photos to delete

[1323] Output: A cleaned-up photo library

[1324] Specific behavior: When the user presses the confirmation button, the photo data to be deleted will be erased from the system.

[1325] Conversational photo search feature

[1326] Step 1:

[1327] The user types or speaks a search query in natural language.

[1328] Input: Natural language search query or voice input

[1329] Output: The search query as text

[1330] What it does: Your smartphone's voice recognition software converts your speech into text in real time.

[1331] Step 2:

[1332] The device converts the voice input into text and sends the text to the server.

[1333] Input: Voice input

[1334] Output: The search query converted to text

[1335] Specific operation: The speech recognition API analyzes the speech, converts it into a string, and then sends it to the server.

[1336] Step 3:

[1337] The server analyzes the search query and searches for matching photos.

[1338] Input: Search query converted to text

[1339] Output: A list of photos from the search results

[1340] How it works: Natural language processing algorithms analyze your query and match it with metadata and features in your photo library to find matching photos.

[1341] Step 4:

[1342] The server sends the discovered photos to the terminal, which displays them to the user.

[1343] Input: Search result photo list

[1344] Output: The photo that is displayed to the user

[1345] Specific operation: The server sends the search result photo data to the terminal, and the terminal visually presents the photo to the user through an interface that displays the photo.

[1346] (Application example 1)

[1347] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1348] There is a demand for a method to efficiently organize, manage, and search large volumes of photos in physical stores and other locations. The goal is to solve these issues by providing an automatic organization function that classifies photos by event or season based on their metadata, a function to remove similar photos, and a function to search for photos using natural language or voice input. It is also important to efficiently analyze user input and provide fast search results.

[1349] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1350] In this invention, the server includes: a means for collecting users who have taken photos and metadata of the photos; a means for classifying photos by event or season based on the collected metadata; a means for automatically creating albums of the classified photos; a means for accepting user voice or text input; a means for converting the input voice into text; a means for analyzing the input using an AI model; a means for searching for relevant photos from the photo library based on the analysis results; and a means for displaying search results to the user. This allows users to easily organize and manage large numbers of photos and quickly find the photos they need. It also improves the accuracy of photo searches and reduces the burden on users.

[1351] "Photo metadata" is information related to a photograph that has been taken, and includes attribute information such as the date and time of the photograph and the location where the photograph was taken.

[1352] "Classifying by event or season" is the process of classifying photos according to specific events (e.g., birthdays, Christmas) or seasons (e.g., spring, summer) based on collected metadata.

[1353] "Automatically create albums" is a function that automatically generates albums according to specific themes or categories based on classified photos.

[1354] "Accepting user voice or text input" refers to a means for recognizing user voice instructions or text-entered search queries.

[1355] "Converting input speech to text" is the process of using speech recognition technology to convert user-entered voice instructions into text format.

[1356] An "AI model" refers to an artificial intelligence algorithm that has been trained to perform tasks such as natural language processing and image recognition.

[1357] "Searching for relevant photos in the photo library" refers to the process of searching for related photos from the photos in the library based on the analyzed data.

[1358] "Displaying search results to the user" refers to the function of displaying the searched photos on the user's device.

[1359] "Photo features" are numerical representations of specific patterns or structures within an image, and are information extracted in the form of facial recognition, hue histograms, etc.

[1360] "Calculating similarity" is a process of numerically evaluating the similarity between photos based on the extracted features.

[1361] "Clustering" refers to an algorithm that groups photos that are highly similar.

[1362] "Presenting photos as candidates for deletion" refers to the process of presenting photos that are deemed unnecessary based on the clustering results to the user and asking for confirmation of deletion.

[1363] "Converting proper nouns, proper names, personal names, and company names into generic names" is the process of replacing specific names or names with more abstract and generic expressions.

[1364] "Noun to superordinate representation" is the process of generalizing a detailed noun representation to its superordinate category.

[1365] "Generating prompt sentences" refers to the process of creating appropriate input sentences for a generative AI model in response to a user query or request.

[1366] The present invention is a system that allows users to efficiently organize, manage, and search for large amounts of photos in a physical store. Specific embodiments of each function are described below.

[1367] Auto-organization function

[1368] The server first collects metadata (such as the date and time of the photo taken, location, etc.) from the photos taken by the device. Based on this metadata, the photos are classified by event or season. Classification involves analyzing timestamps and GPS data. Albums are automatically created based on the classified photos. For example, photos taken in a particular season would be grouped together in a "2023 Summer Campaign" album.

[1369] Conversational photo search feature

[1370] Users can provide a photo search query via voice or text input. For example, a search request might be, "Show me photos of last year's Christmas decorations." The device converts the voice input into text, which is sent to the server. The server uses a generative AI model to analyze the query and search the photo library for matching photos. The search results are sent to the device and displayed to the user.

[1371] Similar photo cleaning function

[1372] The server scans the photos in the photo library and extracts the features of each photo (face recognition, hue histogram, etc.). Based on the extracted features, the similarity between photos is calculated. Photos with high similarity are clustered and presented to the user as candidates for deletion. Photos that the user deems unnecessary are added to the deletion list.

[1373] Explanation of program processing

[1374] The functions of this invention are implemented using smartphones, tablets, and computers as hardware, and the following libraries as software:

[1375] OpenCV: Used for image processing and face recognition.

[1376] scikit-learn: Used for similarity calculations and clustering.

[1377] transformers: Used for NLP processing and generative AI models.

[1378] Specific examples

[1379] Automatic organization: When a store runs a new spring product campaign, related photos are automatically organized into an album called "Spring 2023 New Product Campaign."

[1380] Conversational photo search: When a store manager says, "Show me pictures of last year's Halloween display," the system searches for and displays relevant photos.

[1381] Prompt Sentence Examples

[1382] Below is an example of a prompt sentence that is input to the NLP model.

[1383] "Show me a picture of last year's Christmas decorations."

[1384] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1385] Step 1:

[1386] The device takes a photo and collects its metadata (date, time, location, etc.). The collected metadata is sent to the server. The input is the photo and its metadata, and the output is the metadata sent to the server.

[1387] Step 2:

[1388] The server analyzes the collected metadata and classifies each photo by event or season. For example, photos taken during the summer are classified as a "summer album." The input is the metadata, and the output is the classification result.

[1389] Step 3:

[1390] Based on the classification results, the server automatically creates a photo album. For example, an album titled "2023 Summer Campaign" is generated. The input is the classification results, and the output is the generated album.

[1391] Step 4:

[1392] A user inputs a photo search query in natural language, either by voice or text, such as a request like "Show me pictures of last year's Christmas decorations." The input is the user query, and the output is speech or text data.

[1393] Step 5:

[1394] The device converts voice input into text. Using voice recognition technology, the input voice is converted into text. The converted text is sent to the server. The input is voice data, and the output is text data.

[1395] Step 6:

[1396] The server uses a generative AI model to analyze the input text query. The analysis identifies the conditions for understanding the query and identifying relevant photos. The input is text data, and the output is the analysis results.

[1397] Step 7:

[1398] The server searches for relevant photos in the photo library based on the analysis results. For example, a photo of "last year's Christmas decorations" is searched for based on metadata and features. The input is the analysis results, and the output is the relevant photos.

[1399] Step 8:

[1400] The server sends the search results to the device, which then displays them to the user. The user can view the photos displayed on the screen. The input is the search results, and the output is the displayed photos.

[1401] Step 9:

[1402] It scans photos in your photo library and extracts features for each photo (face recognition, hue histogram, etc.). The input is the photos in your photo library, and the output is the extracted features.

[1403] Step 10:

[1404] The server calculates the similarity between the photos based on the extracted features. For example, it evaluates the similarity numerically using facial recognition or hue information. The input is the features, and the output is the similarity calculation result.

[1405] Step 11:

[1406] Photos with high similarity are clustered and presented to the user as candidates for deletion. Photos that the user deems unnecessary are added to a deletion list. The input is the similarity calculation result, and the output is a presentation of deletion candidates and a deletion list.

[1407] Step 12:

[1408] The user confirms the deletion and the unwanted photos are removed from the photo library. The input is the deletion list and the output is the updated photo library.

[1409] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1410] This invention combines a photo management AI system with an emotion engine that recognizes user emotions, allowing for efficient organization, management, and search of large numbers of photos taken by users, and providing interactions that respond to the user's emotions. Specific embodiments of each function are described below.

[1411] Auto-organization function

[1412] 1. When a device takes a photo, it collects its metadata (date, time, location, etc.).

[1413] 2. Based on the metadata collected by the device, photos are classified by event or season. The classification algorithm analyzes timestamps based on the date and time of the photo and location based on GPS data.

[1414] 3. Your device will automatically create albums based on the classification results. For example, photos taken within a specific period will be grouped into a "Summer Vacation 2023" album.

[1415] Similar photo cleaning function

[1416] 1. The device scans the photos in the photo library and extracts features from each photo (e.g., facial recognition, hue histogram, etc.).

[1417] 2. The device calculates the similarity between the photos based on the extracted features, for example, using a face recognition algorithm.

[1418] 3. The server clusters photos with high similarity and presents deletion candidates to the user. Photos that the user deems unnecessary are added to the deletion list.

[1419] Conversational photo search feature

[1420] 1. A user types or speaks a search query in natural language, for example, "Show me photos from birthday parties in 2022."

[1421] 2. The device converts the voice input into text, which is then sent to the server.

[1422] 3. The server analyzes the search query and searches for matching photos. The server sets search criteria based on the metadata and extracted features.

[1423] 4. The server sends the discovered photos to the device, which displays them to the user.

[1424] Emotion recognition and linking functions using an emotion engine

[1425] 1. The device captures the facial expressions and voice of the user viewing the photo in real time and sends them to the emotion engine.

[1426] 2. The emotion engine analyzes the user's emotional state and identifies emotions such as joy, sadness, surprise, etc.

[1427] 3. The device optimizes the display order of photos according to the user's emotions. For example, if the user is happy, photos of happy memories will be displayed first.

[1428] 4. The emotion engine automatically generates albums based on specific emotional states and selects photos that fit the user's mood. For example, if the user is sad, it will suggest an album of "fun trips."

[1429] Examples:

[1430] Scenario 1: A user returns from a trip and wants to organize their photos.

[1431] 1. The app automatically organizes 100 photos taken by the user on their smartphone while traveling.

[1432] 2. The device creates an album called "Summer Travel 2023" based on the metadata.

[1433] 3. The device scans the photo library, identifies 15 similar photos (the same scene taken from different angles), and presents them as candidates for deletion.

[1434] 4. The user confirms the deletion and the similar photos are deleted.

[1435] Scenario 2: A user wants to show a particular photo to a friend.

[1436] 1. The user says, "Show me pictures of the beach from last summer's vacation."

[1437] 2. The device converts the speech into text and sends it to the server.

[1438] 3. The server analyzes the query and searches for matching photos.

[1439] 4. The server finds the corresponding photo and sends it to the device.

[1440] 5. The device displays the search results to the user.

[1441] Scenario 3: When the user is in a particular emotional state.

[1442] 1. The user smiles while viewing the photo.

[1443] 2. The device sends the user's facial expressions to the emotion engine.

[1444] 3. The emotion engine recognizes the user's emotion as "joy."

[1445] 4. The device will prioritize displaying photos of happy events according to the emotion of "joy."

[1446] 5. The emotion engine automatically generates an album of "happy memories" and suggests them to the user.

[1447] As described above, the system of the present invention significantly improves the efficiency of users' photo management tasks, and by providing interactions that respond to the user's emotions, it realizes a more personalized photo organization, management, and search experience.

[1448] The processing flow will be explained below.

[1449] Auto-organization process steps

[1450] Step 1:

[1451] The user takes a photo.

[1452] Use the camera function of your smartphone to capture a specific scene.

[1453] Step 2:

[1454] The device collects the photo's metadata.

[1455] Obtains the shooting date and time, shooting location (GPS information), camera settings (exposure, ISO, etc.), and scene information.

[1456] Step 3:

[1457] Your device will categorize your photos based on their metadata.

[1458] Analyze photos by event or season based on the date and time of the photo. For example, photos taken within a specific period of time can be classified as "summer events."

[1459] Step 4:

[1460] The device will automatically create albums with categorized photos.

[1461] Create an album called something like "Summer Trip 2023" and add relevant photos to it.

[1462] Similar Photo Cleanup Processing Steps

[1463] Step 1:

[1464] Your device will scan your photo library.

[1465] Load all the photo files and prepare them.

[1466] Step 2:

[1467] The device extracts features from each photo.

[1468] Features are calculated using algorithms such as face recognition, color histogram, and composition analysis.

[1469] Step 3:

[1470] The server calculates the similarity based on the extracted features.

[1471] The cosine similarity and Euclidean distance between feature vectors are calculated to evaluate which photos are similar.

[1472] Step 4:

[1473] The server clusters photos with high similarity.

[1474] Photos with high similarity are grouped into the same cluster.

[1475] Step 5:

[1476] The server presents the deletion candidate list to the user.

[1477] It displays a list of the results and allows the user to choose which photos to delete.

[1478] Step 6:

[1479] The server deletes unwanted photos based on the user's selection.

[1480] Delete unwanted photos from your photo library to free up storage space.

[1481] Conversational Photo Search Processing Steps

[1482] Step 1:

[1483] The user types or speaks a photo search query in natural language.

[1484] For example, type or say, "Show me pictures of Okinawa from last summer vacation."

[1485] Step 2:

[1486] The device converts the speech into text (in the case of voice input).

[1487] A speech recognition algorithm is used to convert instructions into text.

[1488] Step 3:

[1489] The device sends a text query to the server.

[1490] The textual query is sent to the server to make a search request.

[1491] Step 4:

[1492] The server parses the query.

[1493] It uses natural language processing models to interpret the intent of the query and set appropriate search criteria.

[1494] Step 5:

[1495] The server searches the photo library.

[1496] Find photos in your photo library based on the search criteria you set.

[1497] Step 6:

[1498] The server sends the search results to the terminal.

[1499] Sends a list of found photos to your device.

[1500] Step 7:

[1501] The terminal displays the search results to the user.

[1502] Display search results in a gallery view so users can browse photos.

[1503] Processing steps for emotion recognition and linkage functions by the emotion engine

[1504] Step 1:

[1505] The device captures the facial expressions and voice of the user viewing the photos in real time.

[1506] A camera and microphone are used to collect facial expression and voice data of the user.

[1507] Step 2:

[1508] The device sends facial expression data and voice data to the emotion engine.

[1509] The collected data is sent to an emotion engine to analyze the user's emotional state.

[1510] Step 3:

[1511] An emotion engine analyzes the user's emotional state.

[1512] It recognizes the user's emotions by applying algorithms that identify emotions such as joy, sadness, and surprise.

[1513] Step 4:

[1514] The device optimizes the display order of photos according to the user's emotions.

[1515] For example, if the user is happy, photos of happy memories are displayed preferentially.

[1516] Step 5:

[1517] The emotional engine automatically generates albums that correspond to specific emotional states.

[1518] Select photos that match the user's mood and create albums for "happy memories" or "refreshment."

[1519] Step 6:

[1520] The terminal proposes automatically generated albums to the user.

[1521] The app displays an album of "fun trips" and makes suggestions to the user.

[1522] The above is the specific processing flow for each function of the photo management AI system that combines the emotion engine.

[1523] Example 2

[1524] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1525] Conventional photo management systems have the drawback of making it difficult for users to effectively organize, manage, and search large numbers of photos, which is time-consuming. In addition, the photo viewing experience is not personalized based on the user's emotions, which hinders user satisfaction.

[1526] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1527] In this invention, the server includes means for collecting photo metadata, means for classifying photos by event or season, means for automatically creating albums of the classified photos, means for analyzing emotions, means for optimizing the display order of photos based on the emotion analysis results, and means for automatically creating albums according to specific themes based on the emotion analysis. This not only enables efficient organization and management of photos, but also provides a personalized photo viewing experience according to the user's emotions.

[1528] "User" refers to the person taking the photograph and using the system.

[1529] "Metadata" is information related to a photo, including details such as the date and time the photo was taken, the location, and camera settings.

[1530] "Classification" refers to the act of organizing photos into specific events or seasons based on collected metadata.

[1531] An "album" is a group of classified photos, which is automatically created based on a specific theme or period.

[1532] "Emotion analysis means" refers to technology for analyzing and identifying a user's emotional state from data such as facial expressions and voice.

[1533] "Optimizing the display order" refers to the act of adjusting the order of displayed photos based on the results of user sentiment analysis.

[1534] A "themed album" refers to an album that is automatically generated by collecting photos that correspond to a specific emotion or theme based on the results of emotional analysis.

[1535] "Features" are numerically quantified information about the content of a photograph, and refer to data that represents the characteristics of a photograph, such as facial recognition and hue histograms.

[1536] "Similarity" refers to a numerical measure for assessing the content similarity between photographs.

[1537] "Clustering" refers to a technique for grouping similar photos based on extracted features.

[1538] "Candidates for deletion" refers to photos that the user has determined to be unnecessary or that are highly similar and therefore useless.

[1539] "Search query" refers to input instructions, specified by a user in natural language or speech, for searching for a particular photo.

[1540] "Parsing" refers to the process of breaking down the input search query and converting it into appropriate search terms.

[1541] This invention is a photo management AI system that enables efficient organization, management, and search of photos taken by users, and also provides interaction based on the user's emotions. This system collects and analyzes photo metadata, automatically classifies and creates albums, identifies similar photos and suggests deletion options, and optimizes photo display through emotion analysis.

[1542] When a user takes a photo on a device, metadata such as the date and time of the photo and the location are collected. The device then uses the camera API to obtain Exif information and collect the metadata. Based on this metadata, the photos are classified by specific events or seasons. Classification is performed by analyzing the timestamp of the photo and GPS data. This allows an album called, for example, "Summer Trip 2023" to be automatically created.

[1543] Next, the device scans the photos in the photo library and extracts features from each photo. Based on the extracted features, the similarity between photos is calculated. Photos with high similarity are grouped using a clustering algorithm and presented as candidates for deletion. For example, photos of the same scene taken from different angles may be grouped into a specific list.

[1544] With the conversational photo search feature, users enter search queries by voice, and the device converts the speech into text and sends it to the server. The server analyzes the search query, searches for matching photos in the photo library, and displays the results to the user. For example, a user can search for photos using a search query such as "Show me photos of the beach from last summer vacation."

[1545] With the emotion analysis function, the device captures the user's facial expressions and voice in real time and sends them to the emotion engine. The emotion engine uses facial expression recognition algorithms and voice emotion analysis algorithms to analyze the user's emotional state and identify emotions such as joy, sadness, and surprise. This allows the device to optimize the display order of photos according to the user's emotions. For example, if the user smiles while viewing photos, the emotion engine will recognize this as "joy" and prioritize displaying photos of happy events. In addition, the device can automatically generate an album of "happy memories" based on the emotion analysis and suggest them to the user.

[1546] The hardware used includes devices such as smartphones, tablets, and PCs, and the required software includes a camera API, a voice recognition API (such as Google Speech-to-Text API), and an emotion analysis algorithm (such as Face++ or EmoVu). Additionally, server-side search systems such as Elasticsearch and Lucene can be used.

[1547] With the above configuration, this system can significantly improve the efficiency of users' photo management tasks and provide a personalized photo browsing experience based on their emotions. Specific examples of prompts are as follows:

[1548] "Describe an automated photo organization system based on a generative AI model."

[1549] "Please explain with some examples how to manage photos using the emotion engine."

[1550] This allows users to easily search for specific photos and enjoy photos that match their emotions.

[1551] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1552] Step 1:

[1553] Taking photos and collecting metadata

[1554] When a user takes a photo, the device collects metadata in Exif format (such as the date and time the photo was taken, the location, and the camera settings).

[1555] Input: Photo data, metadata at the time of shooting

[1556] Output: A data object containing the photo and its metadata.

[1557] Specific operation: When the smartphone's camera application saves a photo, it simultaneously obtains the Exif information associated with the photo and stores it in an internal database.

[1558] Step 2:

[1559] Classifying photos based on metadata

[1560] The device analyzes the metadata it collects and categorizes photos by event or season based on the date, time, and location they were taken.

[1561] Input: Photo data including metadata

[1562] Output: A list of classified photo data

[1563] What it does: Your device analyzes the timestamps and runs an algorithm to categorize the photos based on events, such as "Summer 2023" or "a specific trip."

[1564] Step 3:

[1565] Auto-create albums

[1566] The device automatically creates an album based on the classification results.

[1567] Input: A list of classified photo data

[1568] Output: Auto-generated album

[1569] Specific operation: The device creates an album named, for example, "Summer Travel 2023," and adds the corresponding photo data to the album.

[1570] Step 4:

[1571] Photo library scanning and feature extraction

[1572] The device scans the photos in your photo library and extracts features such as facial recognition and hue histogram.

[1573] Input: Photo data in the photo library

[1574] Output: Feature data for each photo

[1575] Specific operation: The device uses an image processing library such as OpenCV to extract facial features and color distribution from each photo.

[1576] Step 5:

[1577] Calculating similarity between photos

[1578] The device calculates the similarity between photos based on the features extracted.

[1579] Input: Feature data

[1580] Output: Similarity scores and photo pairs

[1581] Specific operation: The device performs a similarity calculation method (e.g., Euclidean distance) based on face recognition and hue distribution to calculate a similarity score.

[1582] Step 6:

[1583] Clustering and suggesting deletion candidates

[1584] The server clusters photos with high similarity and presents them to the user as candidates for deletion.

[1585] Input: Similarity score and photo pair

[1586] Output: Clustering results and removal candidate list

[1587] What it does: The server uses a clustering algorithm such as DBSCAN to group similar photos and displays deletion candidates to the user on the interface.

[1588] Step 7:

[1589] Confirm and execute the deletion

[1590] The user selects unwanted photos from the presented list of photos to delete and confirms the deletion.

[1591] Input: Deletion candidate list and user selection

[1592] Output: Updated photo library

[1593] What happens: When the user presses the Delete button, the selected photo is deleted from the photo library.

[1594] Step 8:

[1595] Entering search queries in natural language

[1596] The user enters a search query by voice.

[1597] Input: Audio data

[1598] Output: Text query

[1599] Specific action: Say to your voice assistant or smart speaker, "Show me photos of the beach from last summer vacation."

[1600] Step 9:

[1601] Speech-to-text

[1602] The device converts the voice input into text and sends it to the server.

[1603] Input: Audio data

[1604] Output: Text data

[1605] What happens: A speech recognition API (e.g., Google Speech-to-Text API) analyzes the audio data and generates corresponding text.

[1606] Step 10:

[1607] Search query analysis and photo search

[1608] The server analyzes the text query and searches for matching photos in the photo library.

[1609] Input: Text query

[1610] Output: List of photo data

[1611] Specific operation: Analyzes queries using a natural language processing model (generative AI model) and searches for photos that match the criteria based on metadata and features.

[1612] Step 11:

[1613] Displaying search results

[1614] The server sends the search results to the terminal, which displays them to the user.

[1615] Input: List of photo data

[1616] Output: Displaying the photo on a user interface

[1617] Specific operation: The retrieved photos are displayed on the device display in album or thumbnail format.

[1618] Step 12:

[1619] Acquiring and sending emotion data

[1620] The device captures the user's facial expressions and voice in real time and sends them to the emotion engine.

[1621] Input: Real-time facial expression data, voice data

[1622] Output: Data for sentiment analysis

[1623] Specific behavior: Uses a camera and microphone to capture the user's facial expressions and voice in real time and send them to the emotion engine.

[1624] Step 13:

[1625] Sentiment Analysis and Identification

[1626] The emotion engine analyzes the user's emotional state and identifies emotions such as joy, sadness, surprise, etc.

[1627] Input: Facial expression data, voice data

[1628] Output: Emotional state label (e.g., happy, sad)

[1629] Specific operation: Identify emotions using facial expression recognition algorithms and voice emotion analysis algorithms.

[1630] Step 14:

[1631] Optimize the display order of photos

[1632] The device optimizes the display order of photos according to the user's emotions.

[1633] Input: Emotional state labels, photo data from the photo library

[1634] Output: A list of photos in an optimized display order

[1635] What it does: Based on your emotional state, it rearranges the photos in your photo library and prioritizes photos that evoke positive emotions.

[1636] Step 15:

[1637] Emotion-based album generation

[1638] The emotion engine automatically generates albums that correspond to specific emotional states and suggests them to the user.

[1639] Input: Emotional state labels, classified photo data

[1640] Output: Thematic albums

[1641] Specific operation: For example, if the user is sad, an album of photos of "happy memories" is automatically generated and displayed to the user.

[1642] (Application example 2)

[1643] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1644] In physical stores, there is a need to make optimal product recommendations in real time based on customer emotions. However, conventional methods have the problem that it is difficult for store staff to directly observe customers' emotions and recommend appropriate products, which takes time and effort. Furthermore, because it is not possible to accurately grasp customers' emotions, the recommended products often do not meet the customer's needs. To solve these problems, a system is needed that can recognize customers' emotions in real time and recommend products based on them.

[1645] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing customer emotions in real time, means for optimizing photos to be displayed based on the emotion recognition, means for providing interactions based on the emotion recognition, and means for suggesting target information based on the emotion recognition. This makes it possible to accurately grasp customer emotions and suggest optimal products based on that.

[1646] The "user who took the photo" refers to the person who captured the still image using a camera.

[1647] "Means for collecting metadata from photographs" refers to methods for automatically acquiring and storing relevant information such as the date and time a photograph was taken, the location where it was taken, and camera settings.

[1648] "Means for categorizing photos by event or season based on collected metadata" refers to a method for using acquired metadata information to group photos based on a specific time period or specific location.

[1649] "Means for automatically creating albums of classified photos" refers to a method for aggregating classified photos and organizing them into a collection based on a specific theme or event.

[1650] "Means for recognizing a user's emotions in real time" refers to technology that analyzes biometric information such as a user's facial expressions and voice, and identifies a specific emotional state in real time.

[1651] "Means for optimizing photos to be displayed based on emotion recognition" refers to technology that automatically adjusts the order and content of photos to be displayed according to the user's recognized emotion.

[1652] "Means for providing interactions based on emotion recognition" refers to technology that enables a system to automatically provide appropriate responses and actions in accordance with the user's emotional state.

[1653] "Means for suggesting targeted information based on emotion recognition" refers to technology that analyzes a user's emotional state and, based on this, recommends information on optimal products and services to the user.

[1654] This invention relates to a system that efficiently organizes and manages photos taken by users and suggests products based on the user's emotions. This system is particularly effective in brick-and-mortar stores, where store staff can use smart glasses to grasp customers' emotions in real time and make optimal product suggestions.

[1655] System configuration

[1656] The system of the present invention consists of the following major components:

[1657] 1. Smart Glasses

[1658] The smart glasses are equipped with a camera that captures the faces of customers in the store in real time, and also have a display that provides store staff with the necessary information instantly.

[1659] 2. Server

[1660] The server works in conjunction with an emotion recognition engine to analyze customer emotions and generate a list of product suggestions based on the analysis results.

[1661] 3. Emotion Recognition Engine

[1662] It is a software module that analyzes a customer's facial expressions and voice to identify their emotional state, such as joy, sadness, or surprise.

[1663] 4. Product Proposal Module

[1664] It includes an algorithm to suggest optimal products based on the analysis results of the emotion recognition engine.

[1665] Hardware and software used

[1666] Hardware:

[1667] Smart glasses (e.g. Google Glass, Vuzix Blade)

[1668] In-store camera

[1669] software:

[1670] OpenCV: Capture frames from camera stream

[1671] emotion_recognition_module: Detects customer faces from frames and analyzes emotions

[1672] product_recommendation_module: Search for the best product based on analyzed sentiment data

[1673] How it works

[1674] The server acquires the camera stream using OpenCV and captures frames. From these frames, the emotion recognition engine (emotion_recognition_module) analyzes the customer's facial expressions and identifies their emotional state. Based on the identified emotions, the product recommendation module (product_recommendation_module) generates an appropriate product list. The generated list is displayed on the smart glasses display, and store staff receive instructions on how to recommend the most suitable products to the customer.

[1675] Specific examples

[1676] A specific example of use in a physical store is shown below. For example, consider a case where store staff wear smart glasses to make product recommendations. When a customer enters the display area, the camera installed in the smart glasses captures the person's face and recognizes their emotions in real time. For example, if the customer smiles, the glasses will recognize that emotion as "happiness" and suggest a product that matches that emotion (e.g., a luxury sofa).

[1677] Prompt Sentence Examples

[1678] Below is an example of an input prompt for a generative AI model.

[1679] Recognize faces and estimate emotions. Recommend products based on specific emotions.

[1680] import cv2

[1681] import emotion_recognition_module

[1682] import product_recommendation_module

[1683] Get Camera Stream

[1684] camera = cv2.VideoCapture(0)

[1685] Product information list

[1686] products = [

[1687] {"name": "luxury sofa", "emotion": "joy"},

[1688] {"name": "Relax chair", "emotion": "sadness"},

[1689] {"name": "Energetic Lamp", "emotion": "surprise"}

[1690] ]

[1691] while True:

[1692] Capture frames from the camera stream

[1693] ret, frame = camera.read()

[1694] if not ret:

[1695] break

[1696] Detect faces from frames and analyze emotions

[1697] emotions = emotion_recognition_module.detect_emotion(frame)

[1698] for emotion in emotions:

[1699] Search for products that correspond to each emotion

[1700] recommended_products = product_recommendation_module.recommend_products(products, emotion)

[1701] Display on smart glasses (pseudocode)

[1702] smart_glasses_display(recommended_products)

[1703] break

[1704] camera.release()

[1705] cv2.destroyAllWindows()

[1706]

[1707] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1708] Step 1:

[1709] The server acquires the camera stream, specifically capturing video frames in real time from the in-store camera using OpenCV, where the input is the video camera stream and the output is individual video frames.

[1710] Step 2:

[1711] The server uses an emotion recognition engine to detect faces and analyze emotions in video frames. It receives video frames as input and runs a face detection algorithm to identify the location of faces. It then analyzes facial expressions to identify emotions such as happiness, sadness, surprise, etc. The output is the identified emotion.

[1712] Step 3:

[1713] The server sends the analyzed emotion data to a product suggestion module. The product suggestion module receives the emotion data as input and searches for a list of products corresponding to the emotion. For example, if the emotion of joy is identified, it selects "luxury sofa" from the list. The output is a list of products corresponding to the emotion.

[1714] Step 4:

[1715] The server sends the generated product list to the smart glasses. The input is the product list, and the output is the information to be displayed on the smart glasses' display. The smart glasses then provide the appropriate product information to the store staff.

[1716] Step 5:

[1717] The user (store staff) using the terminal selects products to recommend to customers based on the product suggestions displayed on the smart glasses display, making it possible to suggest optimal products to customers in a timely manner.

[1718] Step 6:

[1719] Users (customers) can review the products offered and get more information about the products they are interested in. This interaction is recorded and may be reflected in future recommendations.

[1720] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1721] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1722] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1723] [Fourth embodiment]

[1724] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1725] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1726] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1727] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1728] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1729] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1730] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1731] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1732] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1733] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1734] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1735] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1736] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1737] The present invention is a photo management AI system that efficiently organizes, manages, and searches large numbers of photos taken by users. Specific embodiments of each function are described below.

[1738] Auto-organization function

[1739] 1. When a device takes a photo, it collects its metadata (date, time, location, etc.).

[1740] 2. Based on the metadata collected by the device, photos are classified by event or season. The classification algorithm analyzes timestamps based on the date and time of the photo and location based on GPS data.

[1741] 3. Your device will automatically create albums based on the classification results, for example, grouping photos taken during a specific period into a "Summer Vacation 2023" album.

[1742] Similar photo cleaning function

[1743] 1. The device scans the photos in the photo library and extracts features from each photo (e.g., facial recognition, hue histogram, etc.).

[1744] 2. The device calculates the similarity between the photos based on the extracted features, for example, using a face recognition algorithm.

[1745] 3. The server clusters photos with high similarity and presents deletion candidates to the user. Photos that the user deems unnecessary are added to the deletion list.

[1746] Conversational photo search feature

[1747] 1. A user types or speaks a search query in natural language, for example, "Show me photos from birthday parties in 2022."

[1748] 2. The device converts the voice input into text, which is then sent to the server.

[1749] 3. The server analyzes the search query and searches for matching photos. The server sets search criteria based on the metadata and extracted features.

[1750] 4. The server sends the discovered photos to the device, which displays them to the user.

[1751] Examples:

[1752] Scenario 1: A user returns from a trip and wants to organize their photos.

[1753] 1. The app automatically organizes 100 photos taken by the user on their smartphone while traveling.

[1754] 2. The device creates an album called "Summer Travel 2023" based on the metadata.

[1755] 3. The device scans the photo library, identifies 15 similar photos (the same scene taken from different angles), and presents them as candidates for deletion.

[1756] 4. The user confirms the deletion and the similar photos are deleted.

[1757] Scenario 2: A user wants to show a particular photo to a friend.

[1758] 1. The user says, "Show me pictures of the beach from last summer's vacation."

[1759] 2. The device converts the speech into text and sends it to the server.

[1760] 3. The server analyzes the query and searches for matching photos.

[1761] 4. The server finds the corresponding photo and sends it to the device.

[1762] 5. The device displays the search results to the user.

[1763] As described above, the system of the present invention significantly improves the efficiency of the user's photo management work, and enables the user to instantly find the photos they need.

[1764] The processing flow will be explained below.

[1765] Auto-organization process steps

[1766] Step 1:

[1767] The user takes a photo.

[1768] Use the camera function of your smartphone to capture a specific scene.

[1769] Step 2:

[1770] The device collects the photo's metadata.

[1771] Obtains the shooting date and time, shooting location (GPS information), camera settings (exposure, ISO, etc.), and scene information.

[1772] Step 3:

[1773] Your device will categorize your photos based on their metadata.

[1774] Analyze photos by event or season based on the date and time of the photo. For example, photos taken within a specific period of time can be classified as "summer events."

[1775] Step 4:

[1776] The device will automatically create albums with categorized photos.

[1777] Create an album called something like "Summer Trip 2023" and add relevant photos to it.

[1778] Similar Photo Cleanup Processing Steps

[1779] Step 1:

[1780] Your device will scan your photo library.

[1781] Load all the photo files and prepare them.

[1782] Step 2:

[1783] The device extracts features from each photo.

[1784] Features are calculated using algorithms such as face recognition, color histogram, and composition analysis.

[1785] Step 3:

[1786] The server calculates the similarity based on the extracted features.

[1787] The cosine similarity and Euclidean distance between feature vectors are calculated to evaluate which photos are similar.

[1788] Step 4:

[1789] The server clusters photos with high similarity.

[1790] Photos with high similarity are grouped into the same cluster.

[1791] Step 5:

[1792] The server presents the deletion candidate list to the user.

[1793] It displays a list of the results and allows the user to choose which photos to delete.

[1794] Step 6:

[1795] The server deletes unwanted photos based on the user's selection.

[1796] Delete unwanted photos from your photo library to free up storage space.

[1797] Conversational Photo Search Processing Steps

[1798] Step 1:

[1799] The user types or speaks a photo search query in natural language.

[1800] For example, type or say, "Show me pictures of Okinawa from last summer vacation."

[1801] Step 2:

[1802] The device converts the speech into text (in the case of voice input).

[1803] A speech recognition algorithm is used to convert instructions into text.

[1804] Step 3:

[1805] The device sends a text query to the server.

[1806] The textual query is sent to the server to make a search request.

[1807] Step 4:

[1808] The server parses the query.

[1809] It uses natural language processing models to interpret the intent of the query and set appropriate search criteria.

[1810] Step 5:

[1811] The server searches the photo library.

[1812] Find photos in your photo library based on the search criteria you set.

[1813] Step 6:

[1814] The server sends the search results to the terminal.

[1815] Sends a list of found photos to your device.

[1816] Step 7:

[1817] The terminal displays the search results to the user.

[1818] Display search results in a gallery view so users can browse photos.

[1819] The above is the specific processing flow for each function of the photo management AI system.

[1820] Example 1

[1821] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1822] In today's world, people take a huge number of photos using smartphones and digital cameras, but it is extremely difficult to efficiently organize these photos and quickly search for the ones they need. Furthermore, when there are a large number of similar photos, the task of reviewing and deleting them one by one takes time and effort. To solve these issues, a system is needed that can automatically organize and manage photos and easily delete similar photos.

[1823] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1824] In this invention, the server includes a means for collecting metadata of captured photos, a means for classifying photos by event or season based on the collected metadata, and a means for automatically creating albums of the classified photos. This allows for efficient organization of photos and quick search for specific photos. The server also includes a means for extracting photo features, a means for calculating similarity based on the extracted features, a means for clustering highly similar photos, a means for presenting photos that are candidates for deletion based on the clustering results, and a means for deleting unnecessary photos. This allows for efficient identification of similar photos and easy deletion of unnecessary photos.

[1825] The "user who took the photo" is a user who uses the camera function to acquire a still image.

[1826] A "metadata collection means" is a device or software that has the ability to automatically obtain additional information associated with a photograph, such as the date and time the photograph was taken, the location, and the camera settings.

[1827] A "means for categorizing photos by event or season" is a device or software that has algorithms or functionality that analyzes collected metadata and groups photos based on specific times or events.

[1828] The "means for automatically creating an album of classified photographs" refers to a device or software that has the function of organizing photographs based on the classification results and arranging them into an album format that is easy for the user to view and manage.

[1829] A "means for extracting features" is a device or software that has the function of analyzing important visual features such as facial recognition and hue histograms in photographs and extracting them as numerical data.

[1830] The "means for calculating similarity" is a device or software that has the function of calculating and comparing the similarity between different photographs based on the extracted feature amounts.

[1831] The "clustering means" is a device or software that has an algorithm or function for grouping photos based on the calculated similarity and grouping photos that belong to the same group together.

[1832] The "means for presenting photos as candidates for deletion" refers to a device or software that has the function of displaying to the user photos that may be deemed unnecessary from among the highly similar photos identified by clustering, and suggesting that they be deleted.

[1833] A "means for entering search queries or providing voice commands using natural language" is a device or software that has an interface that allows a user to enter spoken commands to search for photos, or that has the ability to convert voice input into text.

[1834] A "means for searching for relevant photos within a photo library" is a device or software that has the function of identifying matching photos within a photo library using metadata and features based on an analyzed search query.

[1835] The "means for displaying search results to the user" refers to a device or software that has the function of displaying the searched photographs in a form that can be viewed by the user.

[1836] This invention relates to a photo management AI system for efficiently organizing, managing, and searching large numbers of photos taken by users. This system is mainly composed of a terminal and a server, and specific embodiments thereof are described in detail below.

[1837] Auto-organization function

[1838] When a device takes a photo, its metadata (such as the date and time the photo was taken and the location) is automatically collected. This collection is done using the device's camera and GPS functions. Based on the collected metadata, the device analyzes the timestamp and GPS data to classify the photos by event or season. Categorized photos are identified by a classification algorithm within the system, and the device automatically creates an album. For example, an album called "Summer Vacation 2023" is automatically generated.

[1839] Similar photo cleaning function

[1840] The device scans the photos in the photo library and extracts features such as facial recognition and hue histograms. Image processing software, for example, is used to extract the features. The device then calculates the similarity between photos based on the extracted features. A numerical data comparison algorithm is used to calculate this similarity. The server then clusters photos with high similarity and presents them as candidates for deletion. The server also manages the deletion of photos that the user deems unnecessary.

[1841] Conversational photo search feature

[1842] The user types or speaks a search query in natural language. Speech recognition software converts the query into text. The converted text is sent to a server, which analyzes the search query. The server sets search criteria based on metadata and extracted features and searches for matching photos in the photo library. The searched photos are sent from the server to the device, where they are displayed to the user.

[1843] Specific examples

[1844] For example, after a user returns from a trip, 100 photos taken on a smartphone are automatically organized. The device creates an album called "Summer 2023 Trip" based on the metadata. The device then scans the photo library and identifies 15 similar photos (the same scene taken from different angles) and presents them as candidates for deletion. The user can then confirm the deletion and delete the similar photos.

[1845] Or, if a user wants to show a friend a specific photo, they can say, "Show me photos from the beach last summer." The device converts the speech to text and sends the text to the server. The server analyzes the query, finds photos with relevant metadata, and sends those photos to the device. The device then displays the search results to the user.

[1846] This system significantly improves the efficiency of photo management and allows users to quickly find the photos they need. These specific actions allow users to organize and share their memories more easily, without the hassle of daily photo management.

[1847] Prompt Sentence Examples

[1848] 1. "Organize your summer 2022 travel photos"

[1849] 2. "Show me your Halloween pictures from October."

[1850] 3. "Add the latest family photo to my list for deletion."

[1851] 4. "Looking for photos from last year's Christmas party."

[1852] As described above, the present invention provides a system that supports the effective organization, management, and search of photos in a variety of situations.

[1853] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1854] Auto-organization function

[1855] Step 1:

[1856] When a device takes a photo, it automatically collects metadata such as the date, time and location of the photo.

[1857] Input: The photo taken

[1858] Output: Metadata added to photo files (timestamp, GPS information, etc.)

[1859] How it works: When a smartphone's camera takes a photo, the built-in GPS module and timestamp function retrieve the corresponding metadata and embed it in the photo file.

[1860] Step 2:

[1861] Based on the metadata collected by the device, timestamp analysis and GPS data analysis are performed to classify photos by event or season.

[1862] Input: Photo file with metadata

[1863] Output: Photo files with classification tags

[1864] What it does: Metadata analysis algorithms analyze timestamps and GPS data to add classification tags to photos, such as "Summer 2023 Travel."

[1865] Step 3:

[1866] Based on the classification results, the device will automatically create an album called "Summer Vacation 2023."

[1867] Input: Photo files with classification tags

[1868] Output: Auto-generated album folder

[1869] What it does: The photo management software creates folders based on classification tags and moves or copies relevant photos into those folders.

[1870] Similar photo cleaning function

[1871] Step 1:

[1872] The device scans the photos in your photo library and extracts features such as facial recognition and hue histogram.

[1873] Input: All photos in your photo library

[1874] Output: Feature data for each photo

[1875] What it does: The facial recognition algorithm identifies faces in a photo and extracts their features as numerical data. In addition, it calculates a hue histogram and stores that as a feature.

[1876] Step 2:

[1877] The device calculates the similarity between photos based on the extracted features.

[1878] Input: Feature data

[1879] Output: Similarity score between each photo

[1880] What it does: It runs an algorithm that compares numerical data and calculates the similarity between photos using a score between 0 and 1.

[1881] Step 3:

[1882] The server clusters photos with high similarity and presents them as candidates for deletion.

[1883] Input: Similarity score

[1884] Output: List of photos to delete

[1885] What it does: Groups photos based on similarity scores and prompts the user, "Do you want to delete these photos? They're similar."

[1886] Step 4:

[1887] The user confirms the deletion and deletes the unwanted photos.

[1888] Input: List of photos to delete

[1889] Output: A cleaned-up photo library

[1890] Specific behavior: When the user presses the confirmation button, the photo data to be deleted will be erased from the system.

[1891] Conversational photo search feature

[1892] Step 1:

[1893] The user types or speaks a search query in natural language.

[1894] Input: Natural language search query or voice input

[1895] Output: The search query as text

[1896] What it does: Your smartphone's voice recognition software converts your speech into text in real time.

[1897] Step 2:

[1898] The device converts the voice input into text and sends the text to the server.

[1899] Input: Voice input

[1900] Output: The search query converted to text

[1901] Specific operation: The speech recognition API analyzes the speech, converts it into a string, and then sends it to the server.

[1902] Step 3:

[1903] The server analyzes the search query and searches for matching photos.

[1904] Input: Search query converted to text

[1905] Output: A list of photos from the search results

[1906] How it works: Natural language processing algorithms analyze your query and match it with metadata and features in your photo library to find matching photos.

[1907] Step 4:

[1908] The server sends the discovered photos to the terminal, which displays them to the user.

[1909] Input: Search result photo list

[1910] Output: The photo that is displayed to the user

[1911] Specific operation: The server sends the search result photo data to the terminal, and the terminal visually presents the photo to the user through an interface that displays the photo.

[1912] (Application example 1)

[1913] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1914] There is a demand for a method to efficiently organize, manage, and search large volumes of photos in physical stores and other locations. The goal is to solve these issues by providing an automatic organization function that classifies photos by event or season based on their metadata, a function to remove similar photos, and a function to search for photos using natural language or voice input. It is also important to efficiently analyze user input and provide fast search results.

[1915] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1916] In this invention, the server includes: a means for collecting users who have taken photos and metadata of the photos; a means for classifying photos by event or season based on the collected metadata; a means for automatically creating albums of the classified photos; a means for accepting user voice or text input; a means for converting the input voice into text; a means for analyzing the input using an AI model; a means for searching for relevant photos from the photo library based on the analysis results; and a means for displaying search results to the user. This allows users to easily organize and manage large numbers of photos and quickly find the photos they need. It also improves the accuracy of photo searches and reduces the burden on users.

[1917] "Photo metadata" is information related to a photograph that has been taken, and includes attribute information such as the date and time of the photograph and the location where the photograph was taken.

[1918] "Classifying by event or season" is the process of classifying photos according to specific events (e.g., birthdays, Christmas) or seasons (e.g., spring, summer) based on collected metadata.

[1919] "Automatically create albums" is a function that automatically generates albums according to specific themes or categories based on classified photos.

[1920] "Accepting user voice or text input" refers to a means for recognizing user voice instructions or text-entered search queries.

[1921] "Converting input speech to text" is the process of using speech recognition technology to convert user-entered voice instructions into text format.

[1922] An "AI model" refers to an artificial intelligence algorithm that has been trained to perform tasks such as natural language processing and image recognition.

[1923] "Searching for relevant photos in the photo library" refers to the process of searching for related photos from the photos in the library based on the analyzed data.

[1924] "Displaying search results to the user" refers to the function of displaying the searched photos on the user's device.

[1925] "Photo features" are numerical representations of specific patterns or structures within an image, and are information extracted in the form of facial recognition, hue histograms, etc.

[1926] "Calculating similarity" is a process of numerically evaluating the similarity between photos based on the extracted features.

[1927] "Clustering" refers to an algorithm that groups photos that are highly similar.

[1928] "Presenting photos as candidates for deletion" refers to the process of presenting photos that are deemed unnecessary based on the clustering results to the user and asking for confirmation of deletion.

[1929] "Converting proper nouns, proper names, personal names, and company names into generic names" is the process of replacing specific names or names with more abstract and generic expressions.

[1930] "Noun to superordinate representation" is the process of generalizing a detailed noun representation to its superordinate category.

[1931] "Generating prompt sentences" refers to the process of creating appropriate input sentences for a generative AI model in response to a user query or request.

[1932] The present invention is a system that allows users to efficiently organize, manage, and search for large amounts of photos in a physical store. Specific embodiments of each function are described below.

[1933] Auto-organization function

[1934] The server first collects metadata (such as the date and time of the photo taken, location, etc.) from the photos taken by the device. Based on this metadata, the photos are classified by event or season. Classification involves analyzing timestamps and GPS data. Albums are automatically created based on the classified photos. For example, photos taken in a particular season would be grouped together in a "2023 Summer Campaign" album.

[1935] Conversational photo search feature

[1936] Users can provide a photo search query via voice or text input. For example, a search request might be, "Show me photos of last year's Christmas decorations." The device converts the voice input into text, which is sent to the server. The server uses a generative AI model to analyze the query and search the photo library for matching photos. The search results are sent to the device and displayed to the user.

[1937] Similar photo cleaning function

[1938] The server scans the photos in the photo library and extracts the features of each photo (face recognition, hue histogram, etc.). Based on the extracted features, the similarity between photos is calculated. Photos with high similarity are clustered and presented to the user as candidates for deletion. Photos that the user deems unnecessary are added to the deletion list.

[1939] Explanation of program processing

[1940] The functions of this invention are implemented using smartphones, tablets, and computers as hardware, and the following libraries as software:

[1941] OpenCV: Used for image processing and face recognition.

[1942] scikit-learn: Used for similarity calculations and clustering.

[1943] transformers: Used for NLP processing and generative AI models.

[1944] Specific examples

[1945] Automatic organization: When a store runs a new spring product campaign, related photos are automatically organized into an album called "Spring 2023 New Product Campaign."

[1946] Conversational photo search: When a store manager says, "Show me pictures of last year's Halloween display," the system searches for and displays relevant photos.

[1947] Prompt Sentence Examples

[1948] Below is an example of a prompt sentence that is input to the NLP model.

[1949] "Show me a picture of last year's Christmas decorations."

[1950] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1951] Step 1:

[1952] The device takes a photo and collects its metadata (date, time, location, etc.). The collected metadata is sent to the server. The input is the photo and its metadata, and the output is the metadata sent to the server.

[1953] Step 2:

[1954] The server analyzes the collected metadata and classifies each photo by event or season. For example, photos taken during the summer are classified as a "summer album." The input is the metadata, and the output is the classification result.

[1955] Step 3:

[1956] Based on the classification results, the server automatically creates a photo album. For example, an album titled "2023 Summer Campaign" is generated. The input is the classification results, and the output is the generated album.

[1957] Step 4:

[1958] A user inputs a photo search query in natural language, either by voice or text, such as a request like "Show me pictures of last year's Christmas decorations." The input is the user query, and the output is speech or text data.

[1959] Step 5:

[1960] The device converts voice input into text. Using voice recognition technology, the input voice is converted into text. The converted text is sent to the server. The input is voice data, and the output is text data.

[1961] Step 6:

[1962] The server uses a generative AI model to analyze the input text query. The analysis identifies the conditions for understanding the query and identifying relevant photos. The input is text data, and the output is the analysis results.

[1963] Step 7:

[1964] The server searches for relevant photos in the photo library based on the analysis results. For example, a photo of "last year's Christmas decorations" is searched for based on metadata and features. The input is the analysis results, and the output is the relevant photos.

[1965] Step 8:

[1966] The server sends the search results to the device, which then displays them to the user. The user can view the photos displayed on the screen. The input is the search results, and the output is the displayed photos.

[1967] Step 9:

[1968] It scans photos in your photo library and extracts features for each photo (face recognition, hue histogram, etc.). The input is the photos in your photo library, and the output is the extracted features.

[1969] Step 10:

[1970] The server calculates the similarity between the photos based on the extracted features. For example, it evaluates the similarity numerically using facial recognition or hue information. The input is the features, and the output is the similarity calculation result.

[1971] Step 11:

[1972] Photos with high similarity are clustered and presented to the user as candidates for deletion. Photos that the user deems unnecessary are added to a deletion list. The input is the similarity calculation result, and the output is a presentation of deletion candidates and a deletion list.

[1973] Step 12:

[1974] The user confirms the deletion and the unwanted photos are removed from the photo library. The input is the deletion list and the output is the updated photo library.

[1975] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1976] This invention combines a photo management AI system with an emotion engine that recognizes user emotions, allowing for efficient organization, management, and search of large numbers of photos taken by users, and providing interactions that respond to the user's emotions. Specific embodiments of each function are described below.

[1977] Auto-organization function

[1978] 1. When a device takes a photo, it collects its metadata (date, time, location, etc.).

[1979] 2. Based on the metadata collected by the device, photos are classified by event or season. The classification algorithm analyzes timestamps based on the date and time of the photo and location based on GPS data.

[1980] 3. Your device will automatically create albums based on the classification results. For example, photos taken within a specific period will be grouped into a "Summer Vacation 2023" album.

[1981] Similar photo cleaning function

[1982] 1. The device scans the photos in the photo library and extracts features from each photo (e.g., facial recognition, hue histogram, etc.).

[1983] 2. The device calculates the similarity between the photos based on the extracted features, for example, using a face recognition algorithm.

[1984] 3. The server clusters photos with high similarity and presents deletion candidates to the user. Photos that the user deems unnecessary are added to the deletion list.

[1985] Conversational photo search feature

[1986] 1. A user types or speaks a search query in natural language, for example, "Show me photos from birthday parties in 2022."

[1987] 2. The device converts the voice input into text, which is then sent to the server.

[1988] 3. The server analyzes the search query and searches for matching photos. The server sets search criteria based on the metadata and extracted features.

[1989] 4. The server sends the discovered photos to the device, which displays them to the user.

[1990] Emotion recognition and linking functions using an emotion engine

[1991] 1. The device captures the facial expressions and voice of the user viewing the photo in real time and sends them to the emotion engine.

[1992] 2. The emotion engine analyzes the user's emotional state and identifies emotions such as joy, sadness, surprise, etc.

[1993] 3. The device optimizes the display order of photos according to the user's emotions. For example, if the user is happy, photos of happy memories will be displayed first.

[1994] 4. The emotion engine automatically generates albums based on specific emotional states and selects photos that fit the user's mood. For example, if the user is sad, it will suggest an album of "fun trips."

[1995] Examples:

[1996] Scenario 1: A user returns from a trip and wants to organize their photos.

[1997] 1. The app automatically organizes 100 photos taken by the user on their smartphone while traveling.

[1998] 2. The device creates an album called "Summer Travel 2023" based on the metadata.

[1999] 3. The device scans the photo library, identifies 15 similar photos (the same scene taken from different angles), and presents them as candidates for deletion.

[2000] 4. The user confirms the deletion and the similar photos are deleted.

[2001] Scenario 2: A user wants to show a particular photo to a friend.

[2002] 1. The user says, "Show me pictures of the beach from last summer's vacation."

[2003] 2. The device converts the speech into text and sends it to the server.

[2004] 3. The server analyzes the query and searches for matching photos.

[2005] 4. The server finds the corresponding photo and sends it to the device.

[2006] 5. The device displays the search results to the user.

[2007] Scenario 3: When the user is in a particular emotional state.

[2008] 1. The user smiles while viewing the photo.

[2009] 2. The device sends the user's facial expressions to the emotion engine.

[2010] 3. The emotion engine recognizes the user's emotion as "joy."

[2011] 4. The device will prioritize displaying photos of happy events according to the emotion of "joy."

[2012] 5. The emotion engine automatically generates an album of "happy memories" and suggests them to the user.

[2013] As described above, the system of the present invention significantly improves the efficiency of users' photo management tasks, and by providing interactions that respond to the user's emotions, it realizes a more personalized photo organization, management, and search experience.

[2014] The processing flow will be explained below.

[2015] Auto-organization process steps

[2016] Step 1:

[2017] The user takes a photo.

[2018] Use the camera function of your smartphone to capture a specific scene.

[2019] Step 2:

[2020] The device collects the photo's metadata.

[2021] Obtains the shooting date and time, shooting location (GPS information), camera settings (exposure, ISO, etc.), and scene information.

[2022] Step 3:

[2023] Your device will categorize your photos based on their metadata.

[2024] Analyze photos by event or season based on the date and time of the photo. For example, photos taken within a specific period of time can be classified as "summer events."

[2025] Step 4:

[2026] The device will automatically create albums with categorized photos.

[2027] Create an album called something like "Summer Trip 2023" and add relevant photos to it.

[2028] Similar Photo Cleanup Processing Steps

[2029] Step 1:

[2030] Your device will scan your photo library.

[2031] Load all the photo files and prepare them.

[2032] Step 2:

[2033] The device extracts features from each photo.

[2034] Features are calculated using algorithms such as face recognition, color histogram, and composition analysis.

[2035] Step 3:

[2036] The server calculates the similarity based on the extracted features.

[2037] The cosine similarity and Euclidean distance between feature vectors are calculated to evaluate which photos are similar.

[2038] Step 4:

[2039] The server clusters photos with high similarity.

[2040] Photos with high similarity are grouped into the same cluster.

[2041] Step 5:

[2042] The server presents the deletion candidate list to the user.

[2043] It displays a list of the results and allows the user to choose which photos to delete.

[2044] Step 6:

[2045] The server deletes unwanted photos based on the user's selection.

[2046] Delete unwanted photos from your photo library to free up storage space.

[2047] Conversational Photo Search Processing Steps

[2048] Step 1:

[2049] The user types or speaks a photo search query in natural language.

[2050] For example, type or say, "Show me pictures of Okinawa from last summer vacation."

[2051] Step 2:

[2052] The device converts the speech into text (in the case of voice input).

[2053] A speech recognition algorithm is used to convert instructions into text.

[2054] Step 3:

[2055] The device sends a text query to the server.

[2056] The textual query is sent to the server to make a search request.

[2057] Step 4:

[2058] The server parses the query.

[2059] It uses natural language processing models to interpret the intent of the query and set appropriate search criteria.

[2060] Step 5:

[2061] The server searches the photo library.

[2062] Find photos in your photo library based on the search criteria you set.

[2063] Step 6:

[2064] The server sends the search results to the terminal.

[2065] Sends a list of found photos to your device.

[2066] Step 7:

[2067] The terminal displays the search results to the user.

[2068] Display search results in a gallery view so users can browse photos.

[2069] Processing steps for emotion recognition and linkage functions by the emotion engine

[2070] Step 1:

[2071] The device captures the facial expressions and voice of the user viewing the photos in real time.

[2072] A camera and microphone are used to collect facial expression and voice data of the user.

[2073] Step 2:

[2074] The device sends facial expression data and voice data to the emotion engine.

[2075] The collected data is sent to an emotion engine to analyze the user's emotional state.

[2076] Step 3:

[2077] An emotion engine analyzes the user's emotional state.

[2078] It recognizes the user's emotions by applying algorithms that identify emotions such as joy, sadness, and surprise.

[2079] Step 4:

[2080] The device optimizes the display order of photos according to the user's emotions.

[2081] For example, if the user is happy, photos of happy memories are displayed preferentially.

[2082] Step 5:

[2083] The emotional engine automatically generates albums that correspond to specific emotional states.

[2084] Select photos that match the user's mood and create albums for "happy memories" or "refreshment."

[2085] Step 6:

[2086] The terminal proposes automatically generated albums to the user.

[2087] The app displays an album of "fun trips" and makes suggestions to the user.

[2088] The above is the specific processing flow for each function of the photo management AI system that combines the emotion engine.

[2089] Example 2

[2090] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2091] Conventional photo management systems have the drawback of making it difficult for users to effectively organize, manage, and search large numbers of photos, which is time-consuming. In addition, the photo viewing experience is not personalized based on the user's emotions, which hinders user satisfaction.

[2092] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2093] In this invention, the server includes means for collecting photo metadata, means for classifying photos by event or season, means for automatically creating albums of the classified photos, means for analyzing emotions, means for optimizing the display order of photos based on the emotion analysis results, and means for automatically creating albums according to specific themes based on the emotion analysis. This not only enables efficient organization and management of photos, but also provides a personalized photo viewing experience according to the user's emotions.

[2094] "User" refers to the person taking the photograph and using the system.

[2095] "Metadata" is information related to a photo, including details such as the date and time the photo was taken, the location, and camera settings.

[2096] "Classification" refers to the act of organizing photos into specific events or seasons based on collected metadata.

[2097] An "album" is a group of classified photos, which is automatically created based on a specific theme or period.

[2098] "Emotion analysis means" refers to technology for analyzing and identifying a user's emotional state from data such as facial expressions and voice.

[2099] "Optimizing the display order" refers to the act of adjusting the order of displayed photos based on the results of user sentiment analysis.

[2100] A "themed album" refers to an album that is automatically generated by collecting photos that correspond to a specific emotion or theme based on the results of emotional analysis.

[2101] "Features" are numerically quantified information about the content of a photograph, and refer to data that represents the characteristics of a photograph, such as facial recognition and hue histograms.

[2102] "Similarity" refers to a numerical measure for assessing the content similarity between photographs.

[2103] "Clustering" refers to a technique for grouping similar photos based on extracted features.

[2104] "Candidates for deletion" refers to photos that the user has determined to be unnecessary or that are highly similar and therefore useless.

[2105] "Search query" refers to input instructions, specified by a user in natural language or speech, for searching for a particular photo.

[2106] "Parsing" refers to the process of breaking down the input search query and converting it into appropriate search terms.

[2107] This invention is a photo management AI system that enables efficient organization, management, and search of photos taken by users, and also provides interaction based on the user's emotions. This system collects and analyzes photo metadata, automatically classifies and creates albums, identifies similar photos and suggests deletion options, and optimizes photo display through emotion analysis.

[2108] When a user takes a photo on a device, metadata such as the date and time of the photo and the location are collected. The device then uses the camera API to obtain Exif information and collect the metadata. Based on this metadata, the photos are classified by specific events or seasons. Classification is performed by analyzing the timestamp of the photo and GPS data. This allows an album called, for example, "Summer Trip 2023" to be automatically created.

[2109] Next, the device scans the photos in the photo library and extracts features from each photo. Based on the extracted features, the similarity between photos is calculated. Photos with high similarity are grouped using a clustering algorithm and presented as candidates for deletion. For example, photos of the same scene taken from different angles may be grouped into a specific list.

[2110] With the conversational photo search feature, users enter search queries by voice, and the device converts the speech into text and sends it to the server. The server analyzes the search query, searches for matching photos in the photo library, and displays the results to the user. For example, a user can search for photos using a search query such as "Show me photos of the beach from last summer vacation."

[2111] With the emotion analysis function, the device captures the user's facial expressions and voice in real time and sends them to the emotion engine. The emotion engine uses facial expression recognition algorithms and voice emotion analysis algorithms to analyze the user's emotional state and identify emotions such as joy, sadness, and surprise. This allows the device to optimize the display order of photos according to the user's emotions. For example, if the user smiles while viewing photos, the emotion engine will recognize this as "joy" and prioritize displaying photos of happy events. In addition, the device can automatically generate an album of "happy memories" based on the emotion analysis and suggest them to the user.

[2112] The hardware used includes devices such as smartphones, tablets, and PCs, and the required software includes a camera API, a voice recognition API (such as Google Speech-to-Text API), and an emotion analysis algorithm (such as Face++ or EmoVu). Additionally, server-side search systems such as Elasticsearch and Lucene can be used.

[2113] With the above configuration, this system can significantly improve the efficiency of users' photo management tasks and provide a personalized photo browsing experience based on their emotions. Specific examples of prompts are as follows:

[2114] "Describe an automated photo organization system based on a generative AI model."

[2115] "Please explain with some examples how to manage photos using the emotion engine."

[2116] This allows users to easily search for specific photos and enjoy photos that match their emotions.

[2117] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2118] Step 1:

[2119] Taking photos and collecting metadata

[2120] When a user takes a photo, the device collects metadata in Exif format (such as the date and time the photo was taken, the location, and the camera settings).

[2121] Input: Photo data, metadata at the time of shooting

[2122] Output: A data object containing the photo and its metadata.

[2123] Specific operation: When the smartphone's camera application saves a photo, it simultaneously obtains the Exif information associated with the photo and stores it in an internal database.

[2124] Step 2:

[2125] Classifying photos based on metadata

[2126] The device analyzes the metadata it collects and categorizes photos by event or season based on the date, time, and location they were taken.

[2127] Input: Photo data including metadata

[2128] Output: A list of classified photo data

[2129] What it does: Your device analyzes the timestamps and runs an algorithm to categorize the photos based on events, such as "Summer 2023" or "a specific trip."

[2130] Step 3:

[2131] Auto-create albums

[2132] The device automatically creates an album based on the classification results.

[2133] Input: A list of classified photo data

[2134] Output: Auto-generated album

[2135] Specific operation: The device creates an album named, for example, "Summer Travel 2023," and adds the corresponding photo data to the album.

[2136] Step 4:

[2137] Photo library scanning and feature extraction

[2138] The device scans the photos in your photo library and extracts features such as facial recognition and hue histogram.

[2139] Input: Photo data in the photo library

[2140] Output: Feature data for each photo

[2141] Specific operation: The device uses an image processing library such as OpenCV to extract facial features and color distribution from each photo.

[2142] Step 5:

[2143] Calculating similarity between photos

[2144] The device calculates the similarity between photos based on the features extracted.

[2145] Input: Feature data

[2146] Output: Similarity scores and photo pairs

[2147] Specific operation: The device performs a similarity calculation method (e.g., Euclidean distance) based on face recognition and hue distribution to calculate a similarity score.

[2148] Step 6:

[2149] Clustering and suggesting deletion candidates

[2150] The server clusters photos with high similarity and presents them to the user as candidates for deletion.

[2151] Input: Similarity score and photo pair

[2152] Output: Clustering results and removal candidate list

[2153] What it does: The server uses a clustering algorithm such as DBSCAN to group similar photos and displays deletion candidates to the user on the interface.

[2154] Step 7:

[2155] Confirm and execute the deletion

[2156] The user selects unwanted photos from the presented list of photos to delete and confirms the deletion.

[2157] Input: Deletion candidate list and user selection

[2158] Output: Updated photo library

[2159] What happens: When the user presses the Delete button, the selected photo is deleted from the photo library.

[2160] Step 8:

[2161] Entering search queries in natural language

[2162] The user enters a search query by voice.

[2163] Input: Audio data

[2164] Output: Text query

[2165] Specific action: Say to your voice assistant or smart speaker, "Show me photos of the beach from last summer vacation."

[2166] Step 9:

[2167] Speech-to-text

[2168] The device converts the voice input into text and sends it to the server.

[2169] Input: Audio data

[2170] Output: Text data

[2171] What happens: A speech recognition API (e.g., Google Speech-to-Text API) analyzes the audio data and generates corresponding text.

[2172] Step 10:

[2173] Search query analysis and photo search

[2174] The server analyzes the text query and searches for matching photos in the photo library.

[2175] Input: Text query

[2176] Output: List of photo data

[2177] Specific operation: Analyzes queries using a natural language processing model (generative AI model) and searches for photos that match the criteria based on metadata and features.

[2178] Step 11:

[2179] Displaying search results

[2180] The server sends the search results to the terminal, which displays them to the user.

[2181] Input: List of photo data

[2182] Output: Displaying the photo on a user interface

[2183] Specific operation: The retrieved photos are displayed on the device display in album or thumbnail format.

[2184] Step 12:

[2185] Acquiring and sending emotion data

[2186] The device captures the user's facial expressions and voice in real time and sends them to the emotion engine.

[2187] Input: Real-time facial expression data, voice data

[2188] Output: Data for sentiment analysis

[2189] Specific behavior: Uses a camera and microphone to capture the user's facial expressions and voice in real time and send them to the emotion engine.

[2190] Step 13:

[2191] Sentiment Analysis and Identification

[2192] The emotion engine analyzes the user's emotional state and identifies emotions such as joy, sadness, surprise, etc.

[2193] Input: Facial expression data, voice data

[2194] Output: Emotional state label (e.g., happy, sad)

[2195] Specific operation: Identify emotions using facial expression recognition algorithms and voice emotion analysis algorithms.

[2196] Step 14:

[2197] Optimize the display order of photos

[2198] The device optimizes the display order of photos according to the user's emotions.

[2199] Input: Emotional state labels, photo data from the photo library

[2200] Output: A list of photos in an optimized display order

[2201] What it does: Based on your emotional state, it rearranges the photos in your photo library and prioritizes photos that evoke positive emotions.

[2202] Step 15:

[2203] Emotion-based album generation

[2204] The emotion engine automatically generates albums that correspond to specific emotional states and suggests them to the user.

[2205] Input: Emotional state labels, classified photo data

[2206] Output: Thematic albums

[2207] Specific operation: For example, if the user is sad, an album of photos of "happy memories" is automatically generated and displayed to the user.

[2208] (Application example 2)

[2209] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2210] In physical stores, there is a need to make optimal product recommendations in real time based on customer emotions. However, conventional methods have the problem that it is difficult for store staff to directly observe customers' emotions and recommend appropriate products, which takes time and effort. Furthermore, because it is not possible to accurately grasp customers' emotions, the recommended products often do not meet the customer's needs. To solve these problems, a system is needed that can recognize customers' emotions in real time and recommend products based on them.

[2211] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing customer emotions in real time, means for optimizing photos to be displayed based on the emotion recognition, means for providing interactions based on the emotion recognition, and means for suggesting target information based on the emotion recognition. This makes it possible to accurately grasp customer emotions and suggest optimal products based on that.

[2212] The "user who took the photo" refers to the person who captured the still image using a camera.

[2213] "Means for collecting metadata from photographs" refers to methods for automatically acquiring and storing relevant information such as the date and time a photograph was taken, the location where it was taken, and camera settings.

[2214] "Means for categorizing photos by event or season based on collected metadata" refers to a method for using acquired metadata information to group photos based on a specific time period or specific location.

[2215] "Means for automatically creating albums of classified photos" refers to a method for aggregating classified photos and organizing them into a collection based on a specific theme or event.

[2216] "Means for recognizing a user's emotions in real time" refers to technology that analyzes biometric information such as a user's facial expressions and voice, and identifies a specific emotional state in real time.

[2217] "Means for optimizing photos to be displayed based on emotion recognition" refers to technology that automatically adjusts the order and content of photos to be displayed according to the user's recognized emotion.

[2218] "Means for providing interactions based on emotion recognition" refers to technology that enables a system to automatically provide appropriate responses and actions in accordance with the user's emotional state.

[2219] "Means for suggesting targeted information based on emotion recognition" refers to technology that analyzes a user's emotional state and, based on this, recommends information on optimal products and services to the user.

[2220] This invention relates to a system that efficiently organizes and manages photos taken by users and suggests products based on the user's emotions. This system is particularly effective in brick-and-mortar stores, where store staff can use smart glasses to grasp customers' emotions in real time and make optimal product suggestions.

[2221] System configuration

[2222] The system of the present invention consists of the following major components:

[2223] 1. Smart Glasses

[2224] The smart glasses are equipped with a camera that captures the faces of customers in the store in real time, and also have a display that provides store staff with the necessary information instantly.

[2225] 2. Server

[2226] The server works in conjunction with an emotion recognition engine to analyze customer emotions and generate a list of product suggestions based on the analysis results.

[2227] 3. Emotion Recognition Engine

[2228] It is a software module that analyzes a customer's facial expressions and voice to identify their emotional state, such as joy, sadness, or surprise.

[2229] 4. Product Proposal Module

[2230] It includes an algorithm to suggest optimal products based on the analysis results of the emotion recognition engine.

[2231] Hardware and software used

[2232] Hardware:

[2233] Smart glasses (e.g. Google Glass, Vuzix Blade)

[2234] In-store camera

[2235] software:

[2236] OpenCV: Capture frames from camera stream

[2237] emotion_recognition_module: Detects customer faces from frames and analyzes emotions

[2238] product_recommendation_module: Search for the best product based on analyzed sentiment data

[2239] How it works

[2240] The server acquires the camera stream using OpenCV and captures frames. From these frames, the emotion recognition engine (emotion_recognition_module) analyzes the customer's facial expressions and identifies their emotional state. Based on the identified emotions, the product recommendation module (product_recommendation_module) generates an appropriate product list. The generated list is displayed on the smart glasses display, and store staff receive instructions on how to recommend the most suitable products to the customer.

[2241] Specific examples

[2242] A specific example of use in a physical store is shown below. For example, consider a case where store staff wear smart glasses to make product recommendations. When a customer enters the display area, the camera installed in the smart glasses captures the person's face and recognizes their emotions in real time. For example, if the customer smiles, the glasses will recognize that emotion as "happiness" and suggest a product that matches that emotion (e.g., a luxury sofa).

[2243] Prompt Sentence Examples

[2244] Below is an example of an input prompt for a generative AI model.

[2245] Recognize faces and estimate emotions. Recommend products based on specific emotions.

[2246] import cv2

[2247] import emotion_recognition_module

[2248] import product_recommendation_module

[2249] Get Camera Stream

[2250] camera = cv2.VideoCapture(0)

[2251] Product information list

[2252] products = [

[2253] {"name": "luxury sofa", "emotion": "joy"},

[2254] {"name": "Relax chair", "emotion": "sadness"},

[2255] {"name": "Energetic Lamp", "emotion": "surprise"}

[2256] ]

[2257] while True:

[2258] Capture frames from the camera stream

[2259] ret, frame = camera.read()

[2260] if not ret:

[2261] break

[2262] Detect faces from frames and analyze emotions

[2263] emotions = emotion_recognition_module.detect_emotion(frame)

[2264] for emotion in emotions:

[2265] Search for products that correspond to each emotion

[2266] recommended_products = product_recommendation_module.recommend_products(products, emotion)

[2267] Display on smart glasses (pseudocode)

[2268] smart_glasses_display(recommended_products)

[2269] break

[2270] camera.release()

[2271] cv2.destroyAllWindows()

[2272]

[2273] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2274] Step 1:

[2275] The server acquires the camera stream, specifically capturing video frames in real time from the in-store camera using OpenCV, where the input is the video camera stream and the output is individual video frames.

[2276] Step 2:

[2277] The server uses an emotion recognition engine to detect faces and analyze emotions in video frames. It receives video frames as input and runs a face detection algorithm to identify the location of faces. It then analyzes facial expressions to identify emotions such as happiness, sadness, surprise, etc. The output is the identified emotion.

[2278] Step 3:

[2279] The server sends the analyzed emotion data to a product suggestion module. The product suggestion module receives the emotion data as input and searches for a list of products corresponding to the emotion. For example, if the emotion of joy is identified, it selects "luxury sofa" from the list. The output is a list of products corresponding to the emotion.

[2280] Step 4:

[2281] The server sends the generated product list to the smart glasses. The input is the product list, and the output is the information to be displayed on the smart glasses' display. The smart glasses then provide the appropriate product information to the store staff.

[2282] Step 5:

[2283] The user (store staff) using the terminal selects products to recommend to customers based on the product suggestions displayed on the smart glasses display, making it possible to suggest optimal products to customers in a timely manner.

[2284] Step 6:

[2285] Users (customers) can review the products offered and get more information about the products they are interested in. This interaction is recorded and may be reflected in future recommendations.

[2286] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2287] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2288] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2289] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2290] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2291] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2292] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2293] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2294] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2295] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2296] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2297] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2298] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2299] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2300] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2301] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2302] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2303] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2304] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2305] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2306] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2307] The following is further disclosed regarding the above embodiment.

[2308] (Claim 1)

[2309] The user who took the photo,

[2310] a means for collecting metadata of the captured photographs;

[2311] A means to categorize photos by event or season based on collected metadata;

[2312] A means for automatically creating an album of classified photos;

[2313] A system including:

[2314] (Claim 2)

[2315] A means for extracting features of a photo in order to identify similar photos;

[2316] A means for calculating a similarity based on the extracted feature amount;

[2317] A means for clustering photos with high similarity;

[2318] A means for presenting photos as candidates for deletion based on the clustering results;

[2319] a means for deleting unwanted photos after user confirmation;

[2320] 10. The system of claim 1, comprising:

[2321] (Claim 3)

[2322] A means for a user to input or speak a search query using natural language;

[2323] means for parsing an input query;

[2324] A method for searching for relevant photos from within the photo library based on the analysis results,

[2325] a means for displaying search results to a user;

[2326] 10. The system of claim 1, comprising:

[2327] "Example 1"

[2328] (Claim 1)

[2329] The user who took the photo,

[2330] a means for collecting metadata of the captured photographs;

[2331] A means to categorize photos by event or season based on collected metadata;

[2332] A means for automatically creating an album of classified photos;

[2333] A means for generating album folders based on the classified photos;

[2334] A system including:

[2335] (Claim 2)

[2336] A means for extracting features of a photo in order to identify similar photos;

[2337] A means for calculating a similarity based on the extracted feature amount;

[2338] A means for clustering photos with high similarity;

[2339] A means for presenting photos as candidates for deletion based on the clustering results;

[2340] a means for deleting unwanted photos after user confirmation;

[2341] 10. The system of claim 1.

[2342] (Claim 3)

[2343] A means for a user to input or speak a search query using natural language;

[2344] means for parsing an input query;

[2345] A method for searching for relevant photos from within the photo library based on the analysis results,

[2346] a means for displaying search results to a user;

[2347] 10. The system of claim 1.

[2348] "Application Example 1"

[2349] (Claim 1)

[2350] The user who took the photo,

[2351] a means for collecting metadata of the captured photographs;

[2352] A means to categorize photos by event or season based on collected metadata;

[2353] A means for automatically creating an album of classified photos;

[2354] means for accepting user voice or text input;

[2355] a means for converting input speech into text;

[2356] a means for analyzing the input using an AI model;

[2357] A method to search for relevant photos from the photo library based on the analysis results,

[2358] a means for displaying search results to a user;

[2359] A system including:

[2360] (Claim 2)

[2361] A means for extracting features of a photo in order to identify similar photos;

[2362] A means for calculating a similarity based on the extracted feature amount;

[2363] A means for clustering photos with high similarity;

[2364] A means for presenting photos as candidates for deletion based on the clustering results;

[2365] a means for deleting unwanted photos after user confirmation;

[2366] A means to convert proper nouns, proper names, personal names, and company names into general names;

[2367] a means for converting nouns into superordinate expressions;

[2368] 10. The system of claim 1, comprising:

[2369] (Claim 3)

[2370] A means for a user to input or speak a search query using natural language;

[2371] means for parsing an input query;

[2372] A method for searching for relevant photos from within the photo library based on the analysis results,

[2373] a means for displaying search results to a user;

[2374] a means for generating a prompt sentence;

[2375] 10. The system of claim 1, comprising:

[2376] "Example 2: Combining Emotion Engines"

[2377] (Claim 1)

[2378] The user who took the photo,

[2379] a means for collecting metadata of the captured photographs;

[2380] A means to categorize photos by event or season based on collected metadata;

[2381] A means for automatically creating an album of classified photos;

[2382] emotion analysis means for acquiring facial expressions and voices of a user and analyzing their emotions;

[2383] A means for optimizing the display order of photos based on the sentiment analysis results;

[2384] A means for automatically generating albums based on a specific theme based on sentiment analysis;

[2385] A system including:

[2386] (Claim 2)

[2387] A means for extracting features of a photograph;

[2388] A means for calculating a similarity based on the extracted feature amount;

[2389] A means for clustering photos with high similarity;

[2390] A means for presenting photos as candidates for deletion based on the clustering results;

[2391] a means for deleting unwanted photos after user confirmation;

[2392] 10. The system of claim 1, comprising:

[2393] (Claim 3)

[2394] A means for a user to input or speak a search query using natural language;

[2395] means for parsing an input query;

[2396] A method for searching for relevant photos from within the photo library based on the analysis results,

[2397] a means for displaying search results to a user;

[2398] 10. The system of claim 1, comprising:

[2399] "Application example 2 when combining emotion engines"

[2400] (Claim 1)

[2401] The user who took the photo,

[2402] a means for collecting metadata of the captured photographs;

[2403] A means to categorize photos by event or season based on collected metadata;

[2404] A means for automatically creating an album of classified photos;

[2405] means for recognizing user emotions in real time;

[2406] means for optimizing the photos to be displayed based on emotion recognition;

[2407] A means for providing interaction based on emotion recognition;

[2408] means for suggesting target information based on emotion recognition;

[2409] A system including:

[2410] (Claim 2)

[2411] A means for extracting features of a photo in order to identify similar photos;

[2412] A means for calculating a similarity based on...

Claims

1. The user who took the photo, a means for collecting metadata of the captured photographs; A means to categorize photos by event or season based on collected metadata; A means for automatically creating an album of classified photos; A system including:

2. A means for extracting features of a photo in order to identify similar photos; A means for calculating a similarity based on the extracted feature amount; A means for clustering photos with high similarity; A means for presenting photos as candidates for deletion based on the clustering results; a means for deleting unwanted photos after user confirmation; The system of claim 1 , comprising:

3. A means for a user to input or speak a search query using natural language; means for parsing an input query; A method for searching for relevant photos from within the photo library based on the analysis results, a means for displaying search results to a user; The system of claim 1 , comprising:

Images