System

Abstract

A system is provided.SOLUTION: A system comprising: means for uploading images taken by a professional to a cloud storage; means for automatically identifying a person's face from the uploaded images and extracting features of the face; means for identifying a particular person based on the extracted features; means for displaying a list of images of the identified particular person; and means for sending a notification when a new image is uploaded.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Please write the "problem that the invention aims to solve" and "means for solving the problem" in the document in the following format.

[0005] Conventional technologies have the problem that the task of finding a specific person from a large number of images taken by professionals is extremely tedious and likely to overlook the person. In particular, educational institutions such as kindergartens, elementary schools, junior high schools, and high schools take a large number of images for each event, making it difficult and time-consuming for parents to efficiently search for images of their children. It is also not easy to identify a specific person from the images taken and track their growth. Therefore, a new method is needed to automatically classify images and track the growth of a specific person, thereby reducing the effort required for image searches. [Means for solving the problem]

[0006] The present invention provides a means for uploading professionally taken images to cloud storage, automatically identifying people's faces from those images, and extracting their facial features. Furthermore, by providing a means for identifying specific people based on the extracted features and displaying a list of images of that person, the system significantly reduces the effort required for searching. Furthermore, by providing a means for sending notifications when new images are uploaded, the system can immediately notify users of the addition of new images. Furthermore, by adding a means for continuously analyzing accumulated image data to learn the growth process of a specific person, and a means for searching and filtering images based on specific events or dates and times, the system provides a system that allows for more efficient image searches of specific people.

[0007] "Professional" refers to an expert who has advanced skills and knowledge and who performs specific tasks.

[0008] "Image" refers to visual information obtained by a photographic device such as a camera or smartphone.

[0009] "Cloud storage" refers to a service that stores data on remote servers over the Internet and makes it accessible as needed.

[0010] "Human face" refers to the part of an image that contains features such as eyes, nose, and mouth that are necessary to identify a specific person.

[0011] "Identification" refers to the process of analyzing and recognizing the characteristics of a particular person or object in order to distinguish it from others.

[0012] "Feature extraction" refers to the process of extracting specific information or patterns from a target face or object.

[0013] "Specific Person" refers to a specific individual identified by the system.

[0014] "List display" refers to the operation of visually organizing and displaying multiple images or data.

[0015] "Notification" refers to alerts or messages that the system sends to the user to inform them of new information or events.

[0016] "Development" refers to the way a particular person changes and grows over time.

[0017] "Continuous analysis" refers to the operation of continuously analyzing newly added data.

[0018] An "event" is an occurrence that occurs at a specific date, time, and place.

[0019] "Search" refers to the process of locating data based on specific criteria.

[0020] "Filtering" refers to the operation of narrowing down data based on specific conditions.

[0021] A "system" refers to an overall structure in which multiple components work together to perform a specific task or function. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0023] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0026] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0027] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0028] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0030] [First embodiment]

[0031] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0032] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0035] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0038] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0042] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0043] ---

[0044] This invention is a system for automatically classifying images and tracking the growth process of specific individuals. This system uploads professionally taken images to cloud storage, uses an AI engine to identify faces, extracts their features, and displays a list of images of the identified specific individuals.

[0045] System Configuration

[0046] The system mainly consists of the following components:

[0047] User terminal

[0048] server

[0049] Cloud Storage

[0050] AI Engine

[0051] User Registration

[0052] When a user first accesses the application, the user creates an account on the user's device. During this process, the user enters basic information such as an email address, password, child's name, and school year. The server receives this information, validates it, and stores it in the database. This creates a new account and sends the user a notification that the account was created successfully.

[0053] Upload a photo

[0054] Professionally taken images are uploaded to cloud storage. The professional photographer compiles a large number of photos taken at events (e.g., athletic meets, entrance ceremonies, etc.) and adds metadata (event name, date, location, etc.) to each photo. The server receives these images and stores them in cloud storage along with the metadata.

[0055] AI-powered facial recognition and classification

[0056] The server passes the images stored in cloud storage to the AI ​​engine, which uses a face detection algorithm to identify faces in the photos. During this process, facial features (such as eye position, nose shape, and mouth size) are extracted. The server then compares these features with facial data in an existing database and identifies specific people based on the degree of match. The identified images are added to a list of images that are determined to contain specific people.

[0057] Notification function

[0058] When a new image is uploaded to the cloud storage and is recognized by the AI ​​as an image of a specific person, the server sends a notification to the user's device. This notification includes information that a new image has been added. The user's device receives this notification and notifies the user via a pop-up message or other means.

[0059] Viewing and filtering photos

[0060] Users can access the photo browsing screen within the application and set filtering criteria to search for photos based on specific events or dates. The server generates a list of photos based on the user's request and sends it to the user's device, which then visually displays the received photos.

[0061] Learning the process of growth

[0062] The server continuously trains the AI ​​engine based on the accumulated image data. This process tracks the development of specific individuals and improves the accuracy of identification with each new data set. The new data set is used to retrain the facial recognition model, optimizing the overall performance of the system.

[0063] Specific examples

[0064] User Registration

[0065] 1. The user accesses the application and enters the required information on the account creation screen (e.g., Yamada Taro, first grade elementary school student in 2023).

[0066] 2. The server retrieves this information, validates it, and if successful, saves it in the database and creates an account.

[0067] 3. The user device will display a message indicating that the account was created successfully.

[0068] Upload a photo

[0069] 1. A professional photographer will compile photos from the April 2023 entrance ceremony and upload them to cloud storage.

[0070] 2. The server stores these photos in cloud storage and assigns metadata to each photo.

[0071] AI-powered facial recognition and classification

[0072] 1. The server passes the uploaded photo to the AI ​​engine and performs facial recognition.

[0073] 2. The AI ​​engine identifies faces in each photo and extracts features.

[0074] 3. The server identifies a specific person from the database based on the features and adds the image to a list based on the degree of match (e.g., identifying it as a photo of Taro Yamada).

[0075] Notification function

[0076] 1. When the server identifies a new photo as belonging to a specific person, it sends a notification to the user's device.

[0077] 2. The user device displays the message "New photo added."

[0078] Viewing and filtering photos

[0079] 1. The user accesses the photo viewing screen, sets a filter such as "Events: Entrance Ceremony 2023," and performs a search.

[0080] 2. The server searches for the relevant photos and sends the results to the user's device.

[0081] 3. The user terminal displays the search results to the user.

[0082] This allows users to efficiently search for photos based on specific events or dates. Furthermore, as the AI ​​engine continues to learn, its identification accuracy improves, making it easier to track the growth process of a specific person. This system prevents parents from overlooking photos of important events and makes it easier to search for commemorative photos.

[0083] The processing flow will be explained below.

[0084] Handling user registration

[0085] Step 1:

[0086] The user accesses the application and enters the required information (email address, password, child's name, grade, etc.) on the account creation screen.

[0087] Step 2:

[0088] The terminal transmits the input information to the server.

[0089] Step 3:

[0090] The server validates the information received and checks the format and content of the input information.

[0091] Step 4:

[0092] The server saves the validated information to the database and creates the new account.

[0093] Step 5:

[0094] The server notifies the device that the account creation was successful.

[0095] Step 6:

[0096] The device displays a message to the user confirming successful account creation.

[0097] Handling photo uploads

[0098] Step 1:

[0099] A professional photographer takes images from the event and prepares them for upload to cloud storage, where they are tagged with metadata (event name, date, location, etc.).

[0100] Step 2:

[0101] The device sends the image with the metadata to the server.

[0102] Step 3:

[0103] The server stores the received image data in cloud storage.

[0104] AI-powered facial recognition and classification

[0105] Step 1:

[0106] The server prepares the images stored in cloud storage to be passed to the AI ​​engine.

[0107] Step 2:

[0108] The server sends the image data to the AI ​​engine and instructs it to begin facial recognition processing.

[0109] Step 3:

[0110] The AI ​​engine uses a face detection algorithm to identify faces in the image.

[0111] Step 4:

[0112] The AI ​​engine extracts facial features (eye position, nose shape, mouth size, etc.).

[0113] Step 5:

[0114] The server compares the extracted features with face data in an existing database and calculates the degree of match.

[0115] Step 6:

[0116] The server identifies the specific person based on the degree of match and adds them to a photo list of the specific person.

[0117] Handling Notification Functions

[0118] Step 1:

[0119] The server generates a notification when a new image is identified as being of a particular person.

[0120] Step 2:

[0121] The server sends a notification to the user terminal.

[0122] Step 3:

[0123] The device receives the notification and displays the message "New photo added" to the user.

[0124] Photo display and filtering process

[0125] Step 1:

[0126] The user accesses the photo viewing screen and enters search criteria (e.g., event name, date, etc.).

[0127] Step 2:

[0128] The terminal transmits the entered search conditions to the server.

[0129] Step 3:

[0130] Based on the search conditions received by the server, the server searches for corresponding photos from the cloud storage.

[0131] Step 4:

[0132] The server sends the search results to the user terminal.

[0133] Step 5:

[0134] The device receives the search results, organizes them visually, and displays them to the user.

[0135] Processing learning as we grow

[0136] Step 1:

[0137] The server periodically passes the accumulated image data to the AI ​​engine to prepare for retraining.

[0138] Step 2:

[0139] The server sends the AI ​​engine a new image dataset and instructs it to retrain.

[0140] Step 3:

[0141] The AI ​​engine retrains the facial recognition model using the new dataset.

[0142] Step 4:

[0143] The AI ​​engine returns the retrained model to the server.

[0144] Step 5:

[0145] The server uses the improved model for future facial recognition processing.

[0146] These are the specific processing steps of this system, allowing users to quickly and accurately find images of specific people and efficiently track their growth process.

[0147] Example 1

[0148] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0149] Modern digital photo management requires efficient classification of large amounts of photo data and rapid retrieval of photos related to specific people or events. However, manual classification and retrieval is laborious and time-consuming. Improving identification accuracy is particularly important when tracking the growth of specific people. Conventional technologies have not adequately addressed these challenges, and further improvements in the user experience are needed.

[0150] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0151] In this invention, the server includes a means for an end user to input information and create an account, a means for uploading images taken by a professional to network storage, a means for automatically identifying people's faces from the uploaded images and extracting facial features, a means for sending a notification to a user terminal when a new image of a specific person is uploaded, and a means for displaying or filtering the identified images of the specific person based on search conditions. This makes it possible to quickly and accurately search for images related to a specific person or event from a large amount of photo data, and also to track their growth.

[0152] An "end user" is the final user of the system, who performs operations such as uploading, searching, and viewing photos.

[0153] An "expert" is someone who has the skills and knowledge to take photographs and can provide high-quality images.

[0154] "Network storage" is an online storage device accessible via the Internet, and is a system for saving and managing data.

[0155] "Automatic face identification" is the process of using image analysis technology to identify the faces of people present in a photograph.

[0156] "Features" refer to the unique attributes and shape information required to identify a face, and by extracting these, individual people can be recognized.

[0157] A "specific person" is an individual identified based on facial features registered in a database.

[0158] A "user terminal" is a device that connects to the system and operates it, and includes smartphones, tablets, PCs, etc.

[0159] A "notification" is a message from the system to inform the user of new information or events.

[0160] "Search conditions" are filters and keywords that a user sets when searching for a specific photo, and include the event name, date and time, etc.

[0161] "Growth tracking" is the process of analyzing accumulated image data to track the changes in a particular person over the years.

[0162] The present invention provides a system for automatically classifying images and tracking the growth process of a specific person. The system includes a means for an end user to input information and create an account, a means for uploading images taken by a professional to a network storage, a means for automatically identifying faces of people from the uploaded images and extracting facial features, a means for sending a notification to a user terminal when a new image of the specific person is uploaded, and a means for displaying or filtering the identified images of the specific person based on search criteria.

[0163] Hardware and software used

[0164] The system mainly consists of the following components:

[0165] User devices: smartphones, tablets, computers, etc.

[0166] Server: High-performance computing server

[0167] Network storage: Cloud storage services (e.g., Amazon S3 or Google Cloud Storage)

[0168] AI Engine: Machine learning models (e.g., TensorFlow or OpenCV) for face recognition and feature extraction

[0169] Example of user registration

[0170] A user accesses an application and creates an account. For example, the user enters the email address "user@example.com", the password "securepassword", the child's name "Child A", and the grade "First grade of elementary school". The device sends this information to the server, which validates the information. The server then saves the information in the database and a new account is created. The user's device displays a message indicating that the account was created successfully.

[0171] Example of photo upload

[0172] An expert (e.g., a professional photographer) takes photos of an event (e.g., an entrance ceremony) and uploads them to network storage. When uploading, the photos are given metadata such as the event name "Entrance Ceremony 2023," the date "April 1, 2023," and the location "Elementary School A." The server receives these photos and metadata and stores them in network storage.

[0173] Specific examples of AI facial recognition and classification

[0174] The server periodically checks the network storage and detects new photos. It passes the new photos to the AI ​​engine for facial recognition processing. The AI ​​engine uses a face detection algorithm to identify faces in the photos and extract features such as eye position, nose shape, and mouth size. The server compares these features with facial data in an existing database and identifies a specific person (e.g., Child A) based on the degree of match. The server adds the identification results to a list of photos of specific people.

[0175] Examples of notification features

[0176] When the server identifies a new photo of a specific person (e.g., Child A), it sends a notification to the user device saying, "A new photo has been added." The user device receives this notification and notifies the user via a pop-up message or similar.

[0177] Examples of photo display and filtering

[0178] The user accesses the photo viewing screen within the application and sets filtering conditions for a specific event, such as "Entrance Ceremony 2023," or a date, such as "April 1, 2023." The server searches for matching photos based on the user's request and generates results. The user's device displays a list of search results.

[0179] Examples of learning in the process of growth

[0180] The server continuously trains the AI ​​engine based on the accumulated image data. For example, the AI ​​engine retrains based on past photos of Child A, and improves its recognition accuracy by uploading new photos.

[0181] Example prompts for generative AI models

[0182] "Please explain the user registration process in detail, from when the user enters the required information, through when the server validates and saves the information to the database, and notifies the user that the account was successfully created."

[0183] "Please explain in detail the process of uploading photos. Please explain in detail the process from an expert uploading photos to network storage to the server storing them with metadata."

[0184] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0185] Step 1: User Registration

[0186] 1. The user accesses the application and enters the required information (email address, password, child's name, grade, etc.) on the account creation screen.

[0187] Input: Email address "user@example.com", password "securepassword", child's name "Child A", grade "First grade of elementary school"

[0188] Specific operation: The user enters information using the input form and presses the "Register" button.

[0189] 2. The device sends the entered information to the server and requests account creation.

[0190] Output: Data packet containing user information

[0191] Specific operation: The terminal generates a data packet and sends it to the server.

[0192] 3. Validate the information received by the server (check the format of the email address, verify the strength of the password, etc.).

[0193] Input: Data packet containing user information

[0194] Output: Validation result (pass or fail)

[0195] What happens: The server checks the format of the email address and evaluates the strength of the password. If validation is successful, it proceeds to the next step.

[0196] 4. If the server validates successfully, it saves the information in the database and creates a new account.

[0197] Input: User information that has been successfully validated

[0198] Output: Account information stored in the database

[0199] Specific operation: The server opens a database connection and registers the user information as a new record.

[0200] 5. Send a notification to the device that the account was created successfully.

[0201] Input: Account creation success status

[0202] Output: Account creation successful message

[0203] Specific operation: The server generates a success message and sends it to the terminal, which displays the message.

[0204] Step 2: Upload a photo

[0205] 1. Professionals take photos at the event and upload them to network storage.

[0206] Input: Photo file

[0207] Output: Upload request to network storage

[0208] What happens: The expert uses the upload interface to select a photo file and begin uploading.

[0209] 2. An expert assigns metadata to each photo (e.g., event name "Entrance Ceremony 2023", date "April 1, 2023", location "Elementary School A").

[0210] Input: Photo file, metadata

[0211] Output: Photo files with metadata

[0212] Specific operation: The expert enters the required information into the metadata input form and attaches the metadata to the photo.

[0213] 3. The server receives the uploaded photos and metadata and stores them in network storage.

[0214] Input: Photo files with metadata

[0215] Output: Photos stored on network storage

[0216] Specific operation: The server connects to network storage and stores photos and metadata.

[0217] Step 3: AI-powered facial recognition and classification

[0218] 1. The server periodically checks the cloud storage for new photos.

[0219] Input: Photo data from cloud storage

[0220] Output: A list of new photos

[0221] Specific operation: The server sets a timer and periodically accesses the cloud storage to check for new photo files.

[0222] 2. The server passes the new photo to the AI ​​engine for facial recognition processing.

[0223] Input: New photo file

[0224] Output: Face recognition results

[0225] Specific operation: The server passes the photo file to the AI ​​engine, which runs the face detection algorithm.

[0226] 3. The AI ​​engine uses a face detection algorithm to identify faces in the photo and extract their features.

[0227] Input: Photo file

[0228] Output: Facial feature data

[0229] Specific operation: The AI ​​engine detects the face and extracts features such as eye position, nose shape, and mouth size.

[0230] 4. The server compares the features with facial data in an existing database and identifies the person based on the degree of match.

[0231] Input: Facial feature data

[0232] Output: Identification result of a specific person

[0233] Specific operation: The server compares the feature data with face data in the database, calculates the degree of match, and identifies the person.

[0234] 5. The server adds the new photo to the photo list of the identified person.

[0235] Input: Identification result of a specific person

[0236] Output: Updated list of photos of specific people

[0237] Specific operation: The server adds the photo to the list of specific people and updates the list.

[0238] Step 4: Notifications

[0239] 1. The server sends a notification to the user device when a new photo of a specific person is identified.

[0240] Input: Identification results for new photos of a specific person

[0241] Output: Notification message

[0242] Specific operation: The server generates a notification message and sends it to the user terminal.

[0243] 2. The user device receives the notification and notifies the user via a pop-up message or similar.

[0244] Input: Notification message

[0245] Output: Display a popup message

[0246] Specific operation: The user terminal receives the notification message and displays a pop-up message.

[0247] Step 5: View and filter photos

[0248] 1. The user accesses the photo viewing screen within the application and sets filtering criteria for specific events or dates.

[0249] Input: Filtering criteria such as event name and date

[0250] Output: Search request based on filtering criteria

[0251] Specific operation: The user enters conditions into the filtering form and presses the search button.

[0252] 2. The server searches for relevant photos based on the user's request and generates the results.

[0253] Input: Filtering criteria

[0254] Output: A list of photos as search results

[0255] Specific operation: The server searches the photos in the database and generates a list of photos that match the criteria.

[0256] 3. The user device receives the search results and displays a list of photos.

[0257] Input: Photo list as search results

[0258] Output: List of photos

[0259] Specific operation: The user terminal loads the photo list onto the display screen and displays a visual list.

[0260] Step 6: Learning the process of growth

[0261] 1. The server continuously trains the AI ​​engine based on the accumulated image data.

[0262] Input: Stored image data

[0263] Output: Updated AI model

[0264] Specific operation: The server supplies image data to the AI ​​engine and causes it to re-learn.

[0265] 2. The server uses the new dataset to retrain the facial recognition model, improving the system's overall recognition accuracy.

[0266] Input: New dataset, existing AI model

[0267] Output: Retrained AI model

[0268] Specific operation: The server retrains the AI ​​model using a new dataset to improve its recognition accuracy.

[0269] (Application example 1)

[0270] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0271] Providing personalized services to modern consumers is extremely important in increasing customer satisfaction. However, it is difficult to quickly and accurately grasp customer attributes in physical stores and provide services that are appropriate for the customer on the spot. As a result, customer needs cannot be adequately met and store profits are not maximized. It is also difficult to provide a consistent customer experience to multiple customers who visit a store. To solve these issues, a means is needed to use advanced technology to provide personalized services based on customer attributes.

[0272] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0273] In this invention, the server includes means for uploading images taken by a professional to cloud storage, means for automatically identifying people's faces from the uploaded images and extracting their facial features, means for identifying specific people based on the extracted features, means for displaying a list of images of the identified specific people, means for sending a notification when a new image is uploaded, means for identifying the faces of customers visiting the store and specifying the customer's attributes, and means for providing personalized services based on the specified customer attributes. This makes it possible to easily determine customer attributes even in physical stores and provide optimal services to each customer.

[0274] A "professional" is someone who has specialized skills and knowledge and makes a living from them.

[0275] "Cloud storage" refers to remote servers that provide services for storing and managing data over the Internet.

[0276] "Means for identifying human faces" refers to methods and devices that use cameras and image analysis software to identify and recognize human faces in images.

[0277] "Means for extracting facial features" refers to a method of analyzing and acquiring features such as facial shape, pattern, and location information using a facial recognition algorithm.

[0278] The "means for identifying a specific person" refers to a method and apparatus that uses the extracted facial features to match existing facial data in a database and identify matching people.

[0279] "Means for displaying in a list" refers to a method for displaying images of identified specific persons together on a display or screen in a visually easy-to-understand format.

[0280] "Means of sending notifications" refers to the methods of sending messages or alerts to users when new images are uploaded.

[0281] "Means for identifying the faces of customers visiting a store" refers to a method and device for photographing the faces of customers using a camera or other device installed in a physical store and recognizing the faces of customers from the images.

[0282] "Means for identifying customer attributes" refers to a method for analyzing and identifying a customer's age, gender, and other attribute information based on recognized facial features.

[0283] The "means for providing personalized services" refers to a method and apparatus for providing products and services optimized for a customer based on the identified customer's attribute information.

[0284] This invention is a system that identifies customers' faces in brick-and-mortar stores and provides personalized services based on their attributes. The system is mainly composed of a server, cloud storage, an AI engine, and a user's device. The specific configuration and operation of this system are described below.

[0285] User Registration

[0286] When a user first accesses the system through a smartphone application, they create an account. They enter their email address, password, and other basic information, and the server receives, validates, and stores this information in a database.

[0287] Upload a photo

[0288] It provides a way for professionally taken images to be uploaded to cloud storage, where the professional adds metadata such as the event name, date, and location to each photo, and the server receives and stores the metadata in the cloud storage.

[0289] AI-powered facial recognition and classification

[0290] The server passes the images stored in cloud storage to an AI engine for facial identification, which uses a facial recognition algorithm to extract facial features and match them with an existing database to identify a specific person.

[0291] Notification function

[0292] When a new image is uploaded to the cloud storage and identified by the AI ​​engine as belonging to a specific person, the server sends a notification to the user's device, which includes information that a new image has been added.

[0293] Viewing and filtering photos

[0294] Users can access the photo browsing screen in the application and search for photos based on specific events or dates and times. The server searches for the relevant photos based on the user's request and sends them to the user's device.

[0295] Identifying the faces of customers visiting a store

[0296] Using cameras and smartphones installed in physical stores, the server photographs and recognizes the faces of customers visiting the store, extracts their facial features, and uses that information to identify their attributes (age, gender, etc.).

[0297] Providing personalized service

[0298] The server customizes the store's services based on the identified customer attributes, for example, by providing recommended products and specific campaign information.

[0299] Specific examples

[0300] 1. User Registration:

[0301] A user accesses the application and enters basic information.

[0302] The server receives the information and stores it in a database.

[0303] 2. Upload a photo:

[0304] Professionally captured images are uploaded to cloud storage.

[0305] The server receives the images and stores them along with the metadata.

[0306] 3. Identifying the faces of customers visiting your store:

[0307] Cameras installed at the entrances of physical stores capture customers' faces.

[0308] The server analyzes facial features and identifies customer attributes.

[0309] 4. Service Provision:

[0310] The server provides the optimal service to the customer based on the identified customer attribute information.

[0311] Prompt Sentence Examples

[0312] "How can we provide specific services that are best suited to store customers based on facial recognition results? Please provide the following information:

[0313] Facial image

[0314] Customer attributes (age, gender)

[0315] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0316] Step 1:

[0317] A user creates an account through a smartphone application. The user enters their email address, password, and other basic information. The information is sent to the server, where it is validated and stored in a database.

[0318] Input: User's email address, password, basic information

[0319] Data processing: information validation

[0320] Output: Account information is saved in the database

[0321] Step 2:

[0322] Professionally taken images are uploaded to cloud storage, and metadata such as the event name, date, and location are added to the images. The server receives these and stores them in cloud storage.

[0323] Input: Professionally captured images and metadata

[0324] Data Computing: Saving images and metadata to cloud storage

[0325] Output: Images and metadata are saved to cloud storage

[0326] Step 3:

[0327] The server passes the images stored in cloud storage to an AI engine that identifies faces, which uses a facial recognition algorithm to extract facial features and match them with existing databases.

[0328] Input: Images stored in cloud storage

[0329] Data Computing: Facial feature extraction and database matching

[0330] Output: Information about the specific person identified

[0331] Step 4:

[0332] When a new image is uploaded and the AI ​​engine identifies a specific person, the server sends a notification to the user's device, which includes information that a new image has been added.

[0333] Input: Information that identifies a specific person

[0334] Data processing: Notification message generation

[0335] Output: Notification to user terminal

[0336] Step 5:

[0337] The user accesses the photo viewing screen within the application and searches for photos based on a specific event or date and time. The server searches for the corresponding photos based on the user's request and sends them to the user's device.

[0338] Input: User request (event name and date and time)

[0339] Data calculation: Search for relevant photos

[0340] Output: Send search results to the user's terminal

[0341] Step 6:

[0342] A camera installed at the entrance of a physical store captures the faces of customers. The server receives the captured images, analyzes their facial features, and identifies the customer's attributes (age, gender).

[0343] Input: An image of the customer captured by a camera

[0344] Data Computing: Facial feature analysis and attribute identification

[0345] Output: Customer attribute information

[0346] Step 7:

[0347] The server customizes the store's services based on the identified customer attribute information, for example, by recommending specific products or providing specific campaign information.

[0348] Input: Customer attribute information

[0349] Data processing: Creating customized service content

[0350] Output: personalized service presentation

[0351] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0352] ---

[0353] The system uploads professionally taken images to cloud storage, uses an AI engine to identify faces, extracts their features to recognize specific people, and uses an additional emotion engine to identify the user's emotions, thereby personalizing the images displayed. The system consists of the following components:

[0354] User terminal

[0355] server

[0356] Cloud Storage

[0357] AI Engine

[0358] Emotion Engine

[0359] User Registration

[0360] When a user first accesses the application, the user creates an account on the user's device. During this process, the user enters basic information such as an email address, password, child's name, and school year. The server receives this information, validates it, and stores it in the database. This creates a new account and sends the user a notification that the account was created successfully.

[0361] Upload a photo

[0362] Professionally taken images are uploaded to cloud storage. The professional photographer compiles a large number of photos taken at the event and adds metadata (event name, date, location, etc.) to each photo. The server receives these images and stores them in cloud storage along with the metadata.

[0363] AI-powered facial recognition and classification

[0364] The server passes the images stored in cloud storage to the AI ​​engine, which uses a face detection algorithm to identify faces in the photos. During this process, facial features (such as eye position, nose shape, and mouth size) are extracted. The server then compares these features with facial data in an existing database and identifies specific people based on the degree of match. The identified images are added to a list of images that are determined to contain specific people.

[0365] Emotion recognition by emotion engine

[0366] The user device analyzes the user's reaction to the displayed image through an emotion engine, which analyzes the user's facial expressions, tone of voice, text input, etc. to identify emotions (happiness, surprise, sadness, etc.).

[0367] Emotion-based display adjustment

[0368] Based on the perceived emotion, the device will adjust the next image displayed: for example, if the user expresses positive emotion toward a particular image, it will continue to display images with a similar theme, or if negative emotion toward a particular image is detected, it will switch to images with a different theme or person.

[0369] Notification function

[0370] When a new image is uploaded to the cloud storage and is recognized by the AI ​​as an image of a specific person, the server sends a notification to the user's device that a new image has been added, allowing the user to immediately view the new image.

[0371] Viewing and filtering photos

[0372] Users can access the photo browsing screen within the application and set filtering criteria to search for photos based on specific events or dates. The server generates a list of photos based on the user's request and sends it to the user's device, which then visually displays the received photos.

[0373] Learning the process of growth

[0374] The server continuously trains the AI ​​engine based on the accumulated image data. This process tracks the development of specific individuals and improves the accuracy of identification with each new data set. The new data set is used to retrain the facial recognition model, optimizing the overall performance of the system.

[0375] Specific examples

[0376] User Registration

[0377] 1. The user accesses the application and enters the required information on the account creation screen (e.g., Yamada Taro, first grade elementary school student in 2023).

[0378] 2. The server retrieves this information, validates it, and if successful, saves it in the database and creates an account.

[0379] 3. The user device will display a message indicating that the account was created successfully.

[0380] Upload a photo

[0381] 1. A professional photographer will compile photos from the April 2023 entrance ceremony and upload them to cloud storage.

[0382] 2. The server stores these photos in cloud storage and assigns metadata to each photo.

[0383] AI-powered facial recognition and classification

[0384] 1. The server passes the uploaded photo to the AI ​​engine and performs facial recognition.

[0385] 2. The AI ​​engine identifies faces in each photo and extracts features.

[0386] 3. The server identifies a specific person from the database based on the features and adds the image to a list based on the degree of match (e.g., identifying it as a photo of Taro Yamada).

[0387] Emotion recognition by emotion engine

[0388] 1. The emotion engine analyzes the user's reaction to the displayed image (facial expression, tone of voice, text input).

[0389] 2. The emotion engine identifies the user's emotion and sends the data to the server.

[0390] Emotion-based display adjustment

[0391] 1. The server selects the next image to display based on the recognized emotion.

[0392] 2. If a user expresses positive feelings about a particular image, the device is instructed to display images with a similar theme.

[0393] 3. If the user expresses negative emotions, instruct the device to switch to an image of a different subject or person.

[0394] Notification function

[0395] 1. When the server identifies a new photo as belonging to a specific person, it sends a notification to the user's device.

[0396] 2. The user device displays the message "New photo added."

[0397] Viewing and filtering photos

[0398] 1. The user accesses the photo viewing screen, sets a filter such as "Events: Entrance Ceremony 2023," and performs a search.

[0399] 2. The server searches for the relevant photos and sends the results to the user's device.

[0400] 3. The user terminal displays the search results to the user.

[0401] This allows users to efficiently search for photos based on specific events or dates. Furthermore, as the AI ​​engine continues to learn, its identification accuracy improves, making it easier to track the growth process of a specific person. The addition of an emotion engine further improves the user experience by identifying the user's emotions and personalizing the images displayed. This system prevents parents from overlooking photos of important events and makes it easier to search for and display commemorative photos.

[0402] The processing flow will be explained below.

[0403] Handling user registration

[0404] Step 1:

[0405] The user accesses the application and enters the required information (email address, password, child's name, grade, etc.) on the account creation screen.

[0406] Step 2:

[0407] The terminal transmits the input information to the server.

[0408] Step 3:

[0409] The server validates the information received and checks the format and content of the input information.

[0410] Step 4:

[0411] The server saves the validated information to the database and creates the new account.

[0412] Step 5:

[0413] The server notifies the device that the account creation was successful.

[0414] Step 6:

[0415] The device displays a message to the user confirming successful account creation.

[0416] Handling photo uploads

[0417] Step 1:

[0418] A professional photographer prepares images taken at an event (e.g., athletic meet, entrance ceremony, etc.) for uploading to cloud storage.

[0419] Step 2:

[0420] A professional photographer adds metadata (event name, date, location, etc.) to the images.

[0421] Step 3:

[0422] The device sends the image with the metadata to the server.

[0423] Step 4:

[0424] The server stores the received image data in cloud storage.

[0425] AI-powered facial recognition and classification

[0426] Step 1:

[0427] The server prepares the images stored in cloud storage to be passed to the AI ​​engine.

[0428] Step 2:

[0429] The server sends the image data to the AI ​​engine and instructs it to begin facial recognition processing.

[0430] Step 3:

[0431] The AI ​​engine uses a face detection algorithm to identify faces in the image.

[0432] Step 4:

[0433] The AI ​​engine extracts facial features (eye position, nose shape, mouth size, etc.).

[0434] Step 5:

[0435] The server compares the extracted features with face data in an existing database and calculates the degree of match.

[0436] Step 6:

[0437] The server identifies the specific person based on the degree of match and adds them to a photo list of the specific person.

[0438] Emotion recognition processing by emotion engine

[0439] Step 1:

[0440] The user terminal transmits the user's facial expression, tone of voice, text input, etc. in response to the displayed image to the emotion engine.

[0441] Step 2:

[0442] The emotion engine analyzes the user's emotions and generates emotion data (e.g., joy, surprise, sadness, etc.).

[0443] Step 3:

[0444] The emotion data generated by the emotion engine is sent to the server.

[0445] Handling emotion-based display adjustments

[0446] Step 1:

[0447] The server receives the recognized emotion data and selects the next image to display.

[0448] Step 2:

[0449] The server sends the selected image to the terminal.

[0450] Step 3:

[0451] The terminal displays the selected image to the user.

[0452] Handling Notification Functions

[0453] Step 1:

[0454] If the server identifies that a new image has been uploaded to the cloud storage and belongs to a particular person, it generates a notification.

[0455] Step 2:

[0456] The server sends a notification to the user terminal.

[0457] Step 3:

[0458] The device receives the notification and displays the message "New photo added" to the user.

[0459] Photo display and filtering process

[0460] Step 1:

[0461] The user accesses the photo viewing screen and enters search criteria (event name, date, etc.).

[0462] Step 2:

[0463] The terminal sends the search criteria to the server.

[0464] Step 3:

[0465] The server searches for relevant photos from the cloud storage based on the search criteria.

[0466] Step 4:

[0467] The server sends the search results to the user terminal.

[0468] Step 5:

[0469] The terminal receives the search results and displays them to the user.

[0470] Processing learning as we grow

[0471] Step 1:

[0472] The server periodically passes the accumulated image data to the AI ​​engine to prepare for retraining.

[0473] Step 2:

[0474] The server sends the AI ​​engine a new image dataset and instructs it to retrain.

[0475] Step 3:

[0476] The AI ​​engine retrains the facial recognition model using the new dataset.

[0477] Step 4:

[0478] The AI ​​engine returns the retrained model to the server.

[0479] Step 5:

[0480] The server uses the improved model for future facial recognition processing.

[0481] This processing step allows users to efficiently and accurately search for images of specific people and track their progress. Adding an emotion engine also enables personalized image display that reflects the user's emotions.

[0482] Example 2

[0483] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0484] In modern life, there is a need to efficiently manage large amounts of professionally taken images and easily search for images related to specific people or events. However, conventional systems require users to manually review and categorize images individually, which is time-consuming. Furthermore, they do not adjust image display based on the user's emotions, which hinders the quality of the user experience.

[0485] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for uploading images taken by a professional to cloud storage, means for automatically identifying a person's face from the uploaded image and extracting the facial features, and means for identifying a specific person based on the extracted features. This allows a user to automatically identify images of a specific person and display them in a list. The server also includes means for analyzing the user's emotions regarding the displayed image, means for selecting the next image to display based on the analyzed emotional data, and means for sending a notification when a new image is uploaded. This allows the image display to be adjusted according to the user's emotions, providing a more personalized user experience. Furthermore, the server includes means for continuously analyzing accumulated image data to learn the growth process of a specific person and means for searching and filtering images based on specific events and dates and times, allowing users to quickly and easily search for the images they are looking for.

[0486] "Professional" refers to an individual or organization that has specialized knowledge and skills in a particular field and performs work in that field as a profession.

[0487] "Image" means any representation of visual information in digital or analog form, including photographs, illustrations, graphics, etc.

[0488] "Cloud storage" refers to an online storage service for storing, managing, and backing up data over the Internet.

[0489] "Automatic face identification" refers to the process of detecting a human face in an image and identifying that face using a specific algorithm.

[0490] "Extracting facial features" refers to using a face detection algorithm to obtain the characteristic points and patterns of a person's face (e.g., eye position, nose shape, mouth size, etc.) as digital data.

[0491] "Identifying a specific person" refers to the process of identifying a person based on extracted facial features and comparing them with existing data in a database.

[0492] "Analyzing emotions" refers to the process by which the emotion engine identifies a user's emotional state (e.g., joy, surprise, sadness, etc.) based on data such as the user's facial expressions, tone of voice, and text input.

[0493] "Selecting the next image to display based on emotional data" refers to the process of using analyzed emotional data to automatically select and display an image that suits the user.

[0494] "Send notification" means that the system will notify the user through various methods (e.g., push notification, email, text message, etc.) when new images are uploaded to the cloud storage.

[0495] "Continuously analyzing stored image data" refers to periodically evaluating and analyzing stored image data and continuing to analyze it to identify new characteristics or changes in a particular person.

[0496] "Searching and filtering images based on specific events and dates and times" refers to the process of searching the system for relevant images based on specific criteria specified by the user (e.g., event name, date and time, etc.) and displaying only those images that match those criteria.

[0497] This invention relates to a system for efficiently managing a large number of images taken by professionals and adjusting the display according to the user's emotions. The system is composed of the following elements:

[0498] User terminal

[0499] server

[0500] Cloud Storage

[0501] AI Engine

[0502] Emotion Engine

[0503] First, the images taken by the professionals are uploaded to cloud storage. When the user creates an account using their device, the server validates the input information and stores it in the database, allowing the user to access the system.

[0504] The server then sends the images stored in cloud storage to an AI engine, which uses a face detection algorithm to identify people in the images and extract facial features. The server then compares the extracted features with an existing database to identify specific people. The identified images are added to a list and displayed on the user's device.

[0505] As a user browses images, the emotion engine analyzes the user's reactions (facial expressions, tone of voice, and text input) to identify their emotional state. The analyzed emotion data is sent to the server and used to select the next image to be displayed. For example, if a user expresses positive emotion toward a particular image, the server selects an image with a similar theme and instructs the user's device to display it. Conversely, if a negative emotion is recognized, the server selects an image with a different theme or person.

[0506] Additionally, when new images are uploaded to the cloud storage, the server sends a notification to the user's device, which includes a message that a new image has been added, allowing the user to immediately check the new image.

[0507] Other features of the system include a means to continuously analyze the stored image data to learn the developmental progression of a particular person, and the ability to search and filter images based on specific events or dates and times, allowing users to quickly and easily find the images they are looking for.

[0508] A specific example is shown below.

[0509] 1. A user accesses the application, creates an account, and submits their input information (e.g., "Yamada Taro, first grade elementary school student, 2023").

[0510] 2. The server validates the information, stores it in the database, and sends a notification to the user device that the account was successfully created.

[0511] 3. A professional photographer will upload photos from the April 2023 entrance ceremony to cloud storage.

[0512] 4. The server stores these photos in cloud storage and assigns metadata to each photo.

[0513] 5. The server retrieves the photo from the cloud storage and sends it to the AI ​​engine.

[0514] 6. The AI ​​engine identifies the face, extracts features, and compares them with a database to identify a specific person (e.g., "Taro Yamada").

[0515] 7. The emotion engine analyzes the user's reaction and sends the emotion data to the server.

[0516] 8. The server selects the next image to display based on the emotion data and displays it on the user's device.

[0517] 9. When a new image is uploaded, the server sends a notification and the user device displays it.

[0518] 10. The user sets search filters by event or date and time, and the server provides the corresponding images.

[0519] The following is an example of a prompt:

[0520] "Search for and display photos of Taro Yamada's 2023 entrance ceremony."

[0521] The system optimizes the user experience, allowing parents to easily browse photos from important events, and uses an emotion engine to display personalized images based on the user's emotions.

[0522] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0523] Step 1:

[0524] A user accesses an application and creates an account. The specific actions of this step are that the user enters their name, email address, password, and other required information, and clicks the submit button. The entered data (name, email address, password) is sent to the server. The server receives it and performs validation. If validation is successful, the server saves the data in the database and sends a notification to the user terminal that the account was created successfully. The user terminal displays the message "Account created." The input is the user's information, and the output is a notification that the account was created successfully.

[0525] Step 2:

[0526] The professional photographer uploads the photos he has taken to cloud storage. The specific operations of this step are that the professional photographer selects the image files taken for each event and uploads them to a cloud storage service (e.g., Amazon S3). The uploaded images are assigned metadata such as the event name, date, and location. This information is sent to a server. The server receives this information and stores it in cloud storage. The input is the image files and metadata, and the output is the images stored in cloud storage.

[0527] Step 3:

[0528] The server sends the image stored in cloud storage to the AI ​​engine. The specific operation of this step is that the server retrieves the image file from cloud storage and sends it to the AI ​​engine (e.g., Google Cloud Vision API). The input is the image file retrieved from cloud storage, and the output is the face recognition result by the AI ​​engine. The AI ​​engine uses a face detection algorithm to automatically identify the face in the image and extract facial features (e.g., eye position, nose shape, mouth size, etc.).

[0529] Step 4:

[0530] The server receives the feature data sent from the AI ​​engine and compares it with an existing database. The specific operation of this step is that the server compares the feature data with an existing person database and identifies a specific person based on the degree of match. The input is the feature data obtained from the AI ​​engine, and the output is the result of identifying a specific person. The image of the identified person is added to a list and displayed on the user's device.

[0531] Step 5:

[0532] The user device displays the image and sends the user's reaction to the emotion engine. Specific operations of this step include the user device displaying the identified image and sending the user's facial expression, tone of voice, and text input to the emotion engine (e.g., Microsoft Azure Emotion API). The input is the user's reaction data, and the output is the emotion data identified by the emotion engine.

[0533] Step 6:

[0534] The emotion engine analyzes the user's emotions and sends the results to the server. As a specific operation of this step, the emotion engine analyzes the user's reaction data and identifies the emotional state (e.g., joy, surprise, sadness, etc.). The input is the user's reaction data, and the output is the analyzed emotion data. The server receives this data and selects the next image to display.

[0535] Step 7:

[0536] The server selects the next image to display based on the emotion data and instructs the user terminal. As a specific operation of this step, the server generates a list of images to display next based on the analyzed emotion data. For example, if the user expresses positive emotion toward a particular image, the server selects images with a similar theme. The input is the analyzed emotion data, and the output is a list of images to display next. The user terminal displays the images according to this instruction.

[0537] Step 8:

[0538] The server sends a notification when a new image is uploaded. The specific operation of this step is that the server monitors the cloud storage and sends a notification to the user device when a new image is uploaded. The input is the newly uploaded image, and the output is the notification to the user device. The user device displays a notification saying "A new photo has been added."

[0539] Step 9:

[0540] The user sets a search filter based on an event or date and time, and the server provides the relevant images. In this step, the user enters search criteria based on a specific event or date and time within the application and presses the search button. The input is the search filter set by the user, and the output is a list of relevant images. The server searches for relevant images based on this request and sends the results to the user's device. The user's device displays the search results.

[0541] Example prompt sentence:

[0542] "Search and view photos from the 2023 Commencement Ceremony."

[0543] (Application example 2)

[0544] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0545] Existing image management systems are required to efficiently classify images taken by professionals and provide a means to easily search and display the images they need. However, the lack of a function to dynamically change the display content based on user sentiment limits the ability to improve the user experience. Furthermore, it is often difficult for end users to effectively search and filter images based on specific events or dates and times. A new system that solves this problem is needed.

[0546] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading images taken by a professional to cloud storage, means for automatically identifying people's faces from the uploaded images and extracting their facial features, means for identifying specific people based on the extracted features, means for displaying a list of images of the identified specific people, means for sending a notification when a new image is uploaded, means for recognizing a user's facial expression in real time and identifying their emotion, means for adjusting the next image to be displayed based on the identified emotion, means for continuously analyzing accumulated image data to learn the growth process of a specific person, and means for searching and filtering images based on specific events and dates and times. This allows the server to provide display content optimized based on the user's emotions, making it easy for end users to search and filter for the images they need.

[0547] "Professionally captured images" are high-quality images captured using specialized techniques and equipment.

[0548] "Cloud storage" refers to external data storage services accessible over the Internet that allow users to store and manage data remotely.

[0549] "Means for automatically identifying a person's face" is a technology that detects a person's face in an image and automatically recognizes the position and outline of that face.

[0550] "Means for extracting facial features" refers to a technology that analyzes features such as the eyes, nose, and mouth from a person's face and extracts them as data.

[0551] The "means for identifying a specific person" is a technology that recognizes a specific person by comparing the extracted facial features with facial data registered in a database.

[0552] The "means for displaying images in a list" is a technology that visually organizes images related to an identified person and displays them all together on the screen.

[0553] "Means for sending notifications" refers to technology that sends alerts or messages to user terminals when new information is added.

[0554] "Means for recognizing a user's facial expression in real time" refers to a technology that captures the user's face with a camera, instantly analyzes their facial expression, and determines their emotional state.

[0555] The "means for identifying emotions" is a technology that determines the user's emotions (for example, joy, surprise, sadness, etc.) based on analyzed facial expression data.

[0556] The "means for adjusting the next image to be displayed" is a technique for changing the order or content of images to be presented to the user based on the recognized emotion.

[0557] "Means for learning growth processes" refers to technology that analyzes image data accumulated over time to learn about the growth and changes of a specific person.

[0558] "Means for searching and filtering images" refers to a technique for specifying images based on specific conditions (for example, event name or date and time) and extracting images that meet the purpose.

[0559] The system of the present invention efficiently manages images taken by professionals and improves the user experience by dynamically displaying images according to the user's emotions. This system mainly consists of the following elements.

[0560] Server, user terminal, cloud storage, AI engine, and emotion engine.

[0561] 1. User Registration

[0562] When a user first accesses the application, the user creates an account on the user's device. During this process, the user enters basic information. The server receives this information, validates it, and then saves it in the database to create the account.

[0563] 2. Upload a photo

[0564] Professionally taken images are uploaded to cloud storage. The server receives these images and stores them in cloud storage along with metadata (e.g., event name, date, location).

[0565] 3. AI-powered facial recognition and classification

[0566] The server passes the images stored in cloud storage to an AI engine, which uses a face detection algorithm to identify faces in the photos, identifies specific people based on the extracted facial features, and stores the results in a database.

[0567] 4. Emotion Recognition by Emotion Engine

[0568] The user device analyzes the user's reaction to the displayed images in real time through an emotion engine, which analyzes the user's facial expressions and tone of voice to identify emotions.

[0569] 5. Adjusting display based on emotions

[0570] The server then adjusts the next image to be displayed based on the recognized emotion: if a positive emotion is detected, images with related themes are prioritized; if a negative emotion is detected, images with different themes or people are displayed instead.

[0571] 6. Notification function

[0572] When a new image is uploaded to cloud storage and is recognized by the AI ​​engine as an image of a specific person, the server sends a notification to the user's device, allowing the user to immediately view the new image.

[0573] 7. Viewing and filtering photos

[0574] Users can access the photo browsing screen within the application and filter photos based on events or dates. The server generates a list of photos based on the user's request and sends it to the user's device, allowing the user to visually view the photos.

[0575] 8. Learning the process of growth

[0576] The server continuously trains the AI ​​engine based on the stored image data, tracking the development of specific individuals and improving the accuracy of identification with each new data point.

[0577] Hardware and software used

[0578] The following hardware and software are specifically used to implement this system:

[0579] Hardware:

[0580] 1. Smart glasses (e.g., Google Glass)

[0581] 2. Camera

[0582] software:

[0583] 1. OpenCV (for face recognition)

[0584] 2. EmotionRecognizer (emotion recognition library)

[0585] 3. ProductRecommender (product recommendation engine)

[0586] Specific examples

[0587] As a user walks through a virtual store wearing smart glasses, the camera in the smart glasses captures the user's face and analyzes their facial expressions in real time. For example, if a user smiles after looking at a product, the emotion engine will identify that positive emotion and instantly recommend and display relevant products. In this way, it is possible to suggest products tailored to the user's emotional state.

[0588] Prompt example (text format)

[0589] Design a program that allows smart glasses to recognize the user's facial expressions in real time as they walk through a virtual store and display products according to their emotions. Include the following elements:

[0590] 1. Detect faces from the camera stream.

[0591] 2. Uses facial recognition and emotion identification algorithms.

[0592] 3. If positive sentiment is detected, product recommendations are displayed.

[0593] 4. Specify the hardware and software you will be using.

[0594] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0595] Step 1:

[0596] When a user walks through the virtual store, the user device (smart glasses) takes a picture of the user's face in real time using the built-in camera. As input, it takes the video frames captured by the camera and passes them to the next processing step. As output, it obtains the captured video frames.

[0597] Step 2:

[0598] The server receives the video frame and runs a face detection algorithm using OpenCV. The pixel data of the video frame is passed as input, and the area containing the face is identified. The data is then processed to extract the face position and bounding box. The output is a video frame containing face position information.

[0599] Step 3:

[0600] Based on the identified facial area, the server uses EmotionRecognizer to analyze facial expressions and identify emotions. The input is the position information of the face and the pixel data of that area. The data is then processed by analyzing facial features (e.g., eye shape and mouth open / closed degree) and classifying the emotion (happiness, surprise, etc.). The output is the identified emotion data.

[0601] Step 4:

[0602] The server receives the emotion data and adjusts the next image to be displayed based on the identified emotion. The emotion data and metadata of the products in the virtual store are passed as input. The data is then processed to select products that match the emotion data and generate a list of highly relevant products. The output is the selected product list.

[0603] Step 5:

[0604] Based on the product list, the server sends the product information to be displayed next to the user's terminal. The selected product list is passed as input. Data processing involves packaging the product information in a format appropriate for the user's terminal. The packaged product information is obtained as output.

[0605] Step 6:

[0606] The user terminal displays the product on the smart glasses display based on the received product information. The product information sent from the server is passed as input. Specific operations include displaying product images and explanatory text on the display, visually presenting the product to the user. The output is a product display that the user can visually confirm.

[0607] Through the above processing steps, product suggestions based on the user's emotions are realized in real time, and a system that enhances the user experience is constructed.

[0608] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0609] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0610] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0611] [Second embodiment]

[0612] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0613] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0614] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0615] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0616] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0617] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0618] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0619] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0620] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0621] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0622] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0623] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0624] ---

[0625] This invention is a system for automatically classifying images and tracking the growth process of specific individuals. This system uploads professionally taken images to cloud storage, uses an AI engine to identify faces, extracts their features, and displays a list of images of the identified specific individuals.

[0626] System Configuration

[0627] The system mainly consists of the following components:

[0628] User terminal

[0629] server

[0630] Cloud Storage

[0631] AI Engine

[0632] User Registration

[0633] When a user first accesses the application, the user creates an account on the user's device. During this process, the user enters basic information such as an email address, password, child's name, and school year. The server receives this information, validates it, and stores it in the database. This creates a new account and sends the user a notification that the account was created successfully.

[0634] Upload a photo

[0635] Professionally taken images are uploaded to cloud storage. The professional photographer compiles a large number of photos taken at events (e.g., athletic meets, entrance ceremonies, etc.) and adds metadata (event name, date, location, etc.) to each photo. The server receives these images and stores them in cloud storage along with the metadata.

[0636] AI-powered facial recognition and classification

[0637] The server passes the images stored in cloud storage to the AI ​​engine, which uses a face detection algorithm to identify faces in the photos. During this process, facial features (such as eye position, nose shape, and mouth size) are extracted. The server then compares these features with facial data in an existing database and identifies specific people based on the degree of match. The identified images are added to a list of images that are determined to contain specific people.

[0638] Notification function

[0639] When a new image is uploaded to the cloud storage and is recognized by the AI ​​as an image of a specific person, the server sends a notification to the user's device. This notification includes information that a new image has been added. The user's device receives this notification and notifies the user via a pop-up message or other means.

[0640] Viewing and filtering photos

[0641] Users can access the photo browsing screen within the application and set filtering criteria to search for photos based on specific events or dates. The server generates a list of photos based on the user's request and sends it to the user's device, which then visually displays the received photos.

[0642] Learning the process of growth

[0643] The server continuously trains the AI ​​engine based on the accumulated image data. This process tracks the development of specific individuals and improves the accuracy of identification with each new data set. The new data set is used to retrain the facial recognition model, optimizing the overall performance of the system.

[0644] Specific examples

[0645] User Registration

[0646] 1. The user accesses the application and enters the required information on the account creation screen (e.g., Yamada Taro, first grade elementary school student in 2023).

[0647] 2. The server retrieves this information, validates it, and if successful, saves it in the database and creates an account.

[0648] 3. The user device will display a message indicating that the account was created successfully.

[0649] Upload a photo

[0650] 1. A professional photographer will compile photos from the April 2023 entrance ceremony and upload them to cloud storage.

[0651] 2. The server stores these photos in cloud storage and assigns metadata to each photo.

[0652] AI-powered facial recognition and classification

[0653] 1. The server passes the uploaded photo to the AI ​​engine and performs facial recognition.

[0654] 2. The AI ​​engine identifies faces in each photo and extracts features.

[0655] 3. The server identifies a specific person from the database based on the features and adds the image to a list based on the degree of match (e.g., identifying it as a photo of Taro Yamada).

[0656] Notification function

[0657] 1. When the server identifies a new photo as belonging to a specific person, it sends a notification to the user's device.

[0658] 2. The user device displays the message "New photo added."

[0659] Viewing and filtering photos

[0660] 1. The user accesses the photo viewing screen, sets a filter such as "Events: Entrance Ceremony 2023," and performs a search.

[0661] 2. The server searches for the relevant photos and sends the results to the user's device.

[0662] 3. The user terminal displays the search results to the user.

[0663] This allows users to efficiently search for photos based on specific events or dates. Furthermore, as the AI ​​engine continues to learn, its identification accuracy improves, making it easier to track the growth process of a specific person. This system prevents parents from overlooking photos of important events and makes it easier to search for commemorative photos.

[0664] The processing flow will be explained below.

[0665] Handling user registration

[0666] Step 1:

[0667] The user accesses the application and enters the required information (email address, password, child's name, grade, etc.) on the account creation screen.

[0668] Step 2:

[0669] The terminal transmits the input information to the server.

[0670] Step 3:

[0671] The server validates the information received and checks the format and content of the input information.

[0672] Step 4:

[0673] The server saves the validated information to the database and creates the new account.

[0674] Step 5:

[0675] The server notifies the device that the account creation was successful.

[0676] Step 6:

[0677] The device displays a message to the user confirming successful account creation.

[0678] Handling photo uploads

[0679] Step 1:

[0680] A professional photographer takes images from the event and prepares them for upload to cloud storage, where they are tagged with metadata (event name, date, location, etc.).

[0681] Step 2:

[0682] The device sends the image with the metadata to the server.

[0683] Step 3:

[0684] The server stores the received image data in cloud storage.

[0685] AI-powered facial recognition and classification

[0686] Step 1:

[0687] The server prepares the images stored in cloud storage to be passed to the AI ​​engine.

[0688] Step 2:

[0689] The server sends the image data to the AI ​​engine and instructs it to begin facial recognition processing.

[0690] Step 3:

[0691] The AI ​​engine uses a face detection algorithm to identify faces in the image.

[0692] Step 4:

[0693] The AI ​​engine extracts facial features (eye position, nose shape, mouth size, etc.).

[0694] Step 5:

[0695] The server compares the extracted features with face data in an existing database and calculates the degree of match.

[0696] Step 6:

[0697] The server identifies the specific person based on the degree of match and adds them to a list of photos of the specific person.

[0698] Handling Notification Functions

[0699] Step 1:

[0700] The server generates a notification when a new image is identified as being of a particular person.

[0701] Step 2:

[0702] The server sends a notification to the user terminal.

[0703] Step 3:

[0704] The device receives the notification and displays the message "New photo added" to the user.

[0705] Photo display and filtering process

[0706] Step 1:

[0707] The user accesses the photo viewing screen and enters search criteria (e.g., event name, date, etc.).

[0708] Step 2:

[0709] The terminal transmits the entered search conditions to the server.

[0710] Step 3:

[0711] Based on the search conditions received by the server, the server searches for corresponding photos from the cloud storage.

[0712] Step 4:

[0713] The server sends the search results to the user terminal.

[0714] Step 5:

[0715] The device receives the search results, organizes them visually, and displays them to the user.

[0716] Processing learning as we grow

[0717] Step 1:

[0718] The server periodically passes the accumulated image data to the AI ​​engine to prepare for retraining.

[0719] Step 2:

[0720] The server sends the AI ​​engine a new image dataset and instructs it to retrain.

[0721] Step 3:

[0722] The AI ​​engine retrains the facial recognition model using the new dataset.

[0723] Step 4:

[0724] The AI ​​engine returns the retrained model to the server.

[0725] Step 5:

[0726] The server uses the improved model for future facial recognition processing.

[0727] These are the specific processing steps of this system, allowing users to quickly and accurately find images of specific people and efficiently track their growth process.

[0728] Example 1

[0729] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0730] Modern digital photo management requires efficient classification of large amounts of photo data and rapid retrieval of photos related to specific people or events. However, manual classification and retrieval is laborious and time-consuming. Improving identification accuracy is particularly important when tracking the growth of specific people. Conventional technologies have not adequately addressed these challenges, and further improvements in the user experience are needed.

[0731] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0732] In this invention, the server includes a means for an end user to input information and create an account, a means for uploading images taken by a professional to network storage, a means for automatically identifying people's faces from the uploaded images and extracting facial features, a means for sending a notification to a user terminal when a new image of a specific person is uploaded, and a means for displaying or filtering the identified images of the specific person based on search conditions. This makes it possible to quickly and accurately search for images related to a specific person or event from a large amount of photo data, and also to track their growth.

[0733] An "end user" is the final user of the system, who performs operations such as uploading, searching, and viewing photos.

[0734] An "expert" is someone who has the skills and knowledge to take photographs and can provide high-quality images.

[0735] "Network storage" is an online storage device accessible via the Internet, and is a system for saving and managing data.

[0736] "Automatic face identification" is the process of using image analysis technology to identify the faces of people present in a photograph.

[0737] "Features" refer to the unique attributes and shape information required to identify a face, and by extracting these, individual people can be recognized.

[0738] A "specific person" is an individual identified based on facial features registered in a database.

[0739] A "user terminal" is a device that connects to the system and operates it, and includes smartphones, tablets, PCs, etc.

[0740] A "notification" is a message from the system to inform the user of new information or events.

[0741] "Search conditions" are filters and keywords that a user sets when searching for a specific photo, and include the event name, date and time, etc.

[0742] "Growth tracking" is the process of analyzing accumulated image data to track the changes in a particular person over the years.

[0743] The present invention provides a system for automatically classifying images and tracking the growth process of a specific person. The system includes a means for an end user to input information and create an account, a means for uploading images taken by a professional to a network storage, a means for automatically identifying faces of people from the uploaded images and extracting facial features, a means for sending a notification to a user terminal when a new image of the specific person is uploaded, and a means for displaying or filtering the identified images of the specific person based on search criteria.

[0744] Hardware and software used

[0745] The system mainly consists of the following components:

[0746] User devices: smartphones, tablets, computers, etc.

[0747] Server: High-performance computing server

[0748] Network storage: Cloud storage services (e.g., Amazon S3 or Google Cloud Storage)

[0749] AI Engine: Machine learning models (e.g., TensorFlow or OpenCV) for face recognition and feature extraction

[0750] Example of user registration

[0751] A user accesses an application and creates an account. For example, the user enters the email address "user@example.com", the password "securepassword", the child's name "Child A", and the grade "First grade of elementary school". The device sends this information to the server, which validates the information. The server then saves the information in the database and a new account is created. The user's device displays a message indicating that the account was created successfully.

[0752] Example of photo upload

[0753] An expert (e.g., a professional photographer) takes photos of an event (e.g., an entrance ceremony) and uploads them to network storage. When uploading, the photos are given metadata such as the event name "Entrance Ceremony 2023," the date "April 1, 2023," and the location "Elementary School A." The server receives these photos and metadata and stores them in network storage.

[0754] Specific examples of AI facial recognition and classification

[0755] The server periodically checks the network storage and detects new photos. It passes the new photos to the AI ​​engine for facial recognition processing. The AI ​​engine uses a face detection algorithm to identify faces in the photos and extract features such as eye position, nose shape, and mouth size. The server compares these features with facial data in an existing database and identifies a specific person (e.g., Child A) based on the degree of match. The server adds the identification results to a list of photos of specific people.

[0756] Examples of notification features

[0757] When the server identifies a new photo of a specific person (e.g., Child A), it sends a notification to the user device saying, "A new photo has been added." The user device receives this notification and notifies the user via a pop-up message or similar.

[0758] Examples of photo display and filtering

[0759] The user accesses the photo viewing screen within the application and sets filtering conditions for a specific event, such as "Entrance Ceremony 2023," or a date, such as "April 1, 2023." The server searches for matching photos based on the user's request and generates results. The user's device displays a list of search results.

[0760] Examples of learning in the process of growth

[0761] The server continuously trains the AI ​​engine based on the accumulated image data. For example, the AI ​​engine retrains based on past photos of Child A, and improves its recognition accuracy by uploading new photos.

[0762] Example prompts for generative AI models

[0763] "Please explain the user registration process in detail, from when the user enters the required information, through when the server validates and saves the information to the database, and notifies the user that the account was successfully created."

[0764] "Please explain in detail the process of uploading photos. Please explain in detail the process from an expert uploading photos to network storage to the server storing them with metadata."

[0765] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0766] Step 1: User Registration

[0767] 1. The user accesses the application and enters the required information (email address, password, child's name, grade, etc.) on the account creation screen.

[0768] Input: Email address "user@example.com", password "securepassword", child's name "Child A", grade "First grade of elementary school"

[0769] Specific operation: The user enters information using the input form and presses the "Register" button.

[0770] 2. The device sends the entered information to the server and requests account creation.

[0771] Output: Data packet containing user information

[0772] Specific operation: The terminal generates a data packet and sends it to the server.

[0773] 3. Validate the information received by the server (check the format of the email address, verify the strength of the password, etc.).

[0774] Input: Data packet containing user information

[0775] Output: Validation result (pass or fail)

[0776] What happens: The server checks the format of the email address and evaluates the strength of the password. If validation is successful, it proceeds to the next step.

[0777] 4. If the server validates successfully, it saves the information in the database and creates a new account.

[0778] Input: User information that has been successfully validated

[0779] Output: Account information stored in the database

[0780] Specific operation: The server opens a database connection and registers the user information as a new record.

[0781] 5. Send a notification to the device that the account was created successfully.

[0782] Input: Account creation success status

[0783] Output: Account creation successful message

[0784] Specific operation: The server generates a success message and sends it to the terminal, which displays the message.

[0785] Step 2: Upload a photo

[0786] 1. Professionals take photos at the event and upload them to network storage.

[0787] Input: Photo file

[0788] Output: Upload request to network storage

[0789] What happens: The expert uses the upload interface to select a photo file and begin uploading.

[0790] 2. An expert assigns metadata to each photo (e.g., event name "Entrance Ceremony 2023", date "April 1, 2023", location "Elementary School A").

[0791] Input: Photo file, metadata

[0792] Output: Photo files with metadata

[0793] Specific operation: The expert enters the required information into the metadata input form and attaches the metadata to the photo.

[0794] 3. The server receives the uploaded photos and metadata and stores them in network storage.

[0795] Input: Photo files with metadata

[0796] Output: Photos stored on network storage

[0797] Specific operation: The server connects to network storage and stores photos and metadata.

[0798] Step 3: AI-powered facial recognition and classification

[0799] 1. The server periodically checks the cloud storage for new photos.

[0800] Input: Photo data from cloud storage

[0801] Output: A list of new photos

[0802] Specific operation: The server sets a timer and periodically accesses the cloud storage to check for new photo files.

[0803] 2. The server passes the new photo to the AI ​​engine for facial recognition processing.

[0804] Input: New photo file

[0805] Output: Face recognition results

[0806] Specific operation: The server passes the photo file to the AI ​​engine, which runs the face detection algorithm.

[0807] 3. The AI ​​engine uses a face detection algorithm to identify faces in the photo and extract their features.

[0808] Input: Photo file

[0809] Output: Facial feature data

[0810] Specific operation: The AI ​​engine detects the face and extracts features such as eye position, nose shape, and mouth size.

[0811] 4. The server compares the features with facial data in an existing database and identifies the person based on the degree of match.

[0812] Input: Facial feature data

[0813] Output: Identification result of a specific person

[0814] Specific operation: The server compares the feature data with face data in the database, calculates the degree of match, and identifies the person.

[0815] 5. The server adds the new photo to the photo list of the identified person.

[0816] Input: Identification result of a specific person

[0817] Output: Updated list of photos of specific people

[0818] Specific operation: The server adds the photo to the list of specific people and updates the list.

[0819] Step 4: Notifications

[0820] 1. The server sends a notification to the user device when a new photo of a specific person is identified.

[0821] Input: Identification results for new photos of a specific person

[0822] Output: Notification message

[0823] Specific operation: The server generates a notification message and sends it to the user terminal.

[0824] 2. The user device receives the notification and notifies the user via a pop-up message or similar.

[0825] Input: Notification message

[0826] Output: Display a popup message

[0827] Specific operation: The user terminal receives the notification message and displays a pop-up message.

[0828] Step 5: View and filter photos

[0829] 1. The user accesses the photo viewing screen within the application and sets filtering criteria for specific events or dates.

[0830] Input: Filtering criteria such as event name and date

[0831] Output: Search request based on filtering criteria

[0832] Specific operation: The user enters conditions into the filtering form and presses the search button.

[0833] 2. The server searches for relevant photos based on the user's request and generates the results.

[0834] Input: Filtering criteria

[0835] Output: A list of photos as search results

[0836] Specific operation: The server searches the photos in the database and generates a list of photos that match the criteria.

[0837] 3. The user device receives the search results and displays a list of photos.

[0838] Input: Photo list as search results

[0839] Output: List of photos

[0840] Specific operation: The user terminal loads the photo list onto the display screen and displays a visual list.

[0841] Step 6: Learning the process of growth

[0842] 1. The server continuously trains the AI ​​engine based on the accumulated image data.

[0843] Input: Stored image data

[0844] Output: Updated AI model

[0845] Specific operation: The server supplies image data to the AI ​​engine and causes it to re-learn.

[0846] 2. The server uses the new dataset to retrain the facial recognition model, improving the system's overall recognition accuracy.

[0847] Input: New dataset, existing AI model

[0848] Output: Retrained AI model

[0849] Specific operation: The server retrains the AI ​​model using a new dataset to improve its recognition accuracy.

[0850] (Application example 1)

[0851] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0852] Providing personalized services to modern consumers is extremely important in increasing customer satisfaction. However, it is difficult to quickly and accurately grasp customer attributes in physical stores and provide services that are appropriate for the customer on the spot. As a result, customer needs cannot be adequately met and store profits are not maximized. It is also difficult to provide a consistent customer experience to multiple customers who visit a store. To solve these issues, a means is needed to use advanced technology to provide personalized services based on customer attributes.

[0853] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0854] In this invention, the server includes means for uploading images taken by a professional to cloud storage, means for automatically identifying people's faces from the uploaded images and extracting their facial features, means for identifying specific people based on the extracted features, means for displaying a list of images of the identified specific people, means for sending a notification when a new image is uploaded, means for identifying the faces of customers visiting the store and specifying the customer's attributes, and means for providing personalized services based on the specified customer attributes. This makes it possible to easily determine customer attributes even in physical stores and provide optimal services to each customer.

[0855] A "professional" is someone who has specialized skills and knowledge and makes a living from them.

[0856] "Cloud storage" refers to remote servers that provide services for storing and managing data over the Internet.

[0857] "Means for identifying human faces" refers to methods and devices that use cameras and image analysis software to identify and recognize human faces in images.

[0858] "Means for extracting facial features" refers to a method of analyzing and acquiring features such as facial shape, pattern, and location information using a facial recognition algorithm.

[0859] The "means for identifying a specific person" refers to a method and apparatus that uses the extracted facial features to match existing facial data in a database and identify matching people.

[0860] "Means for displaying in a list" refers to a method for displaying images of identified specific persons together on a display or screen in a visually easy-to-understand format.

[0861] "Means of sending notifications" refers to the methods of sending messages or alerts to users when new images are uploaded.

[0862] "Means for identifying the faces of customers visiting a store" refers to a method and device for photographing the faces of customers using a camera or other device installed in a physical store and recognizing the faces of customers from the images.

[0863] "Means for identifying customer attributes" refers to a method for analyzing and identifying a customer's age, gender, and other attribute information based on recognized facial features.

[0864] The "means for providing personalized services" refers to a method and apparatus for providing products and services optimized for a customer based on the identified customer's attribute information.

[0865] This invention is a system that identifies customers' faces in brick-and-mortar stores and provides personalized services based on their attributes. The system is mainly composed of a server, cloud storage, an AI engine, and a user's device. The specific configuration and operation of this system are described below.

[0866] User Registration

[0867] When a user first accesses the system through a smartphone application, they create an account. They enter their email address, password, and other basic information, and the server receives, validates, and stores this information in a database.

[0868] Upload a photo

[0869] It provides a way for professionally taken images to be uploaded to cloud storage, where the professional adds metadata such as the event name, date, and location to each photo, and the server receives and stores the metadata in the cloud storage.

[0870] AI-powered facial recognition and classification

[0871] The server passes the images stored in cloud storage to an AI engine for facial identification, which uses a facial recognition algorithm to extract facial features and match them with an existing database to identify a specific person.

[0872] Notification function

[0873] When a new image is uploaded to the cloud storage and identified by the AI ​​engine as belonging to a specific person, the server sends a notification to the user's device, which includes information that a new image has been added.

[0874] Viewing and filtering photos

[0875] Users can access the photo browsing screen in the application and search for photos based on specific events or dates and times. The server searches for the relevant photos based on the user's request and sends them to the user's device.

[0876] Identifying the faces of customers visiting a store

[0877] Using cameras and smartphones installed in physical stores, the server photographs and recognizes the faces of customers visiting the store, extracts their facial features, and uses that information to identify their attributes (age, gender, etc.).

[0878] Providing personalized service

[0879] The server customizes the store's services based on the identified customer attributes, for example, by providing recommended products and specific campaign information.

[0880] Specific examples

[0881] 1. User Registration:

[0882] A user accesses the application and enters basic information.

[0883] The server receives the information and stores it in a database.

[0884] 2. Upload a photo:

[0885] Professionally captured images are uploaded to cloud storage.

[0886] The server receives the images and stores them along with the metadata.

[0887] 3. Identifying the faces of customers visiting your store:

[0888] Cameras installed at the entrances of physical stores capture customers' faces.

[0889] The server analyzes facial features and identifies customer attributes.

[0890] 4. Service Provision:

[0891] The server provides the optimal service to the customer based on the identified customer attribute information.

[0892] Prompt Sentence Examples

[0893] "How can we provide specific services that are best suited to store customers based on facial recognition results? Please provide the following information:

[0894] Facial image

[0895] Customer attributes (age, gender)

[0896] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0897] Step 1:

[0898] A user creates an account through a smartphone application. The user enters their email address, password, and other basic information. The information is sent to the server, where it is validated and stored in a database.

[0899] Input: User's email address, password, basic information

[0900] Data processing: information validation

[0901] Output: Account information is saved in the database

[0902] Step 2:

[0903] Professionally taken images are uploaded to cloud storage, and metadata such as the event name, date, and location are added to the images. The server receives these and stores them in cloud storage.

[0904] Input: Professionally captured images and metadata

[0905] Data Computing: Saving images and metadata to cloud storage

[0906] Output: Images and metadata are saved to cloud storage

[0907] Step 3:

[0908] The server passes the images stored in cloud storage to an AI engine that identifies faces, which uses a facial recognition algorithm to extract facial features and match them with existing databases.

[0909] Input: Images stored in cloud storage

[0910] Data Computing: Facial feature extraction and database matching

[0911] Output: Information about the specific person identified

[0912] Step 4:

[0913] When a new image is uploaded and the AI ​​engine identifies a specific person, the server sends a notification to the user's device, which includes information that a new image has been added.

[0914] Input: Information that identifies a specific person

[0915] Data processing: Notification message generation

[0916] Output: Notification to user terminal

[0917] Step 5:

[0918] The user accesses the photo viewing screen within the application and searches for photos based on a specific event or date and time. The server searches for the corresponding photos based on the user's request and sends them to the user's device.

[0919] Input: User request (event name and date and time)

[0920] Data calculation: Search for relevant photos

[0921] Output: Send search results to the user's terminal

[0922] Step 6:

[0923] A camera installed at the entrance of a physical store captures the faces of customers. The server receives the captured images, analyzes their facial features, and identifies the customer's attributes (age, gender).

[0924] Input: An image of the customer captured by a camera

[0925] Data Computing: Facial feature analysis and attribute identification

[0926] Output: Customer attribute information

[0927] Step 7:

[0928] The server customizes the store's services based on the identified customer attribute information, for example, by recommending specific products or providing specific campaign information.

[0929] Input: Customer attribute information

[0930] Data processing: Creating customized service content

[0931] Output: personalized service presentation

[0932] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0933] ---

[0934] The system uploads professionally taken images to cloud storage, uses an AI engine to identify faces, extracts their features to recognize specific people, and uses an additional emotion engine to identify the user's emotions, thereby personalizing the images displayed. The system consists of the following components:

[0935] User terminal

[0936] server

[0937] Cloud Storage

[0938] AI Engine

[0939] Emotion Engine

[0940] User Registration

[0941] When a user first accesses the application, the user creates an account on the user's device. During this process, the user enters basic information such as an email address, password, child's name, and school year. The server receives this information, validates it, and stores it in the database. This creates a new account and sends the user a notification that the account was created successfully.

[0942] Upload a photo

[0943] Professionally taken images are uploaded to cloud storage. The professional photographer compiles a large number of photos taken at the event and adds metadata (event name, date, location, etc.) to each photo. The server receives these images and stores them in cloud storage along with the metadata.

[0944] AI-powered facial recognition and classification

[0945] The server passes the images stored in cloud storage to the AI ​​engine, which uses a face detection algorithm to identify faces in the photos. During this process, facial features (such as eye position, nose shape, and mouth size) are extracted. The server then compares these features with facial data in an existing database and identifies specific people based on the degree of match. The identified images are added to a list of images that are determined to contain specific people.

[0946] Emotion recognition by emotion engine

[0947] The user device analyzes the user's reaction to the displayed image through an emotion engine, which analyzes the user's facial expressions, tone of voice, text input, etc. to identify emotions (happiness, surprise, sadness, etc.).

[0948] Emotion-based display adjustment

[0949] Based on the perceived emotion, the device will adjust the next image displayed: for example, if the user expresses positive emotion toward a particular image, it will continue to display images with a similar theme, or if negative emotion toward a particular image is detected, it will switch to images with a different theme or person.

[0950] Notification function

[0951] When a new image is uploaded to the cloud storage and is recognized by the AI ​​as an image of a specific person, the server sends a notification to the user's device that a new image has been added, allowing the user to immediately view the new image.

[0952] Viewing and filtering photos

[0953] Users can access the photo browsing screen within the application and set filtering criteria to search for photos based on specific events or dates. The server generates a list of photos based on the user's request and sends it to the user's device, which then visually displays the received photos.

[0954] Learning the process of growth

[0955] The server continuously trains the AI ​​engine based on the accumulated image data. This process tracks the development of specific individuals and improves the accuracy of identification with each new data set. The new data set is used to retrain the facial recognition model, optimizing the overall performance of the system.

[0956] Specific examples

[0957] User Registration

[0958] 1. The user accesses the application and enters the required information on the account creation screen (e.g., Yamada Taro, first grade elementary school student in 2023).

[0959] 2. The server retrieves this information, validates it, and if successful, saves it in the database and creates an account.

[0960] 3. The user device will display a message indicating that the account was created successfully.

[0961] Upload a photo

[0962] 1. A professional photographer will compile photos from the April 2023 entrance ceremony and upload them to cloud storage.

[0963] 2. The server stores these photos in cloud storage and assigns metadata to each photo.

[0964] AI-powered facial recognition and classification

[0965] 1. The server passes the uploaded photo to the AI ​​engine and performs facial recognition.

[0966] 2. The AI ​​engine identifies faces in each photo and extracts features.

[0967] 3. The server identifies a specific person from the database based on the features and adds the image to a list based on the degree of match (e.g., identifying it as a photo of Taro Yamada).

[0968] Emotion recognition by emotion engine

[0969] 1. The emotion engine analyzes the user's reaction to the displayed image (facial expression, tone of voice, text input).

[0970] 2. The emotion engine identifies the user's emotion and sends the data to the server.

[0971] Emotion-based display adjustment

[0972] 1. The server selects the next image to display based on the recognized emotion.

[0973] 2. If a user expresses positive feelings about a particular image, the device is instructed to display images with a similar theme.

[0974] 3. If the user expresses negative emotions, instruct the device to switch to an image of a different subject or person.

[0975] Notification function

[0976] 1. When the server identifies a new photo as belonging to a specific person, it sends a notification to the user's device.

[0977] 2. The user device displays the message "New photo added."

[0978] Viewing and filtering photos

[0979] 1. The user accesses the photo viewing screen, sets a filter such as "Events: Entrance Ceremony 2023," and performs a search.

[0980] 2. The server searches for the relevant photos and sends the results to the user's device.

[0981] 3. The user terminal displays the search results to the user.

[0982] This allows users to efficiently search for photos based on specific events or dates. Furthermore, as the AI ​​engine continues to learn, its identification accuracy improves, making it easier to track the growth process of a specific person. The addition of an emotion engine further improves the user experience by identifying the user's emotions and personalizing the images displayed. This system prevents parents from overlooking photos of important events and makes it easier to search for and display commemorative photos.

[0983] The processing flow will be explained below.

[0984] Handling user registration

[0985] Step 1:

[0986] The user accesses the application and enters the required information (email address, password, child's name, grade, etc.) on the account creation screen.

[0987] Step 2:

[0988] The terminal transmits the input information to the server.

[0989] Step 3:

[0990] The server validates the information received and checks the format and content of the input information.

[0991] Step 4:

[0992] The server saves the validated information to the database and creates the new account.

[0993] Step 5:

[0994] The server notifies the device that the account creation was successful.

[0995] Step 6:

[0996] The device displays a message to the user confirming successful account creation.

[0997] Handling photo uploads

[0998] Step 1:

[0999] A professional photographer prepares images taken at an event (e.g., athletic meet, entrance ceremony, etc.) for uploading to cloud storage.

[1000] Step 2:

[1001] A professional photographer adds metadata (event name, date, location, etc.) to the images.

[1002] Step 3:

[1003] The device sends the image with the metadata to the server.

[1004] Step 4:

[1005] The server stores the received image data in cloud storage.

[1006] AI-powered facial recognition and classification

[1007] Step 1:

[1008] The server prepares the images stored in cloud storage to be passed to the AI ​​engine.

[1009] Step 2:

[1010] The server sends the image data to the AI ​​engine and instructs it to begin facial recognition processing.

[1011] Step 3:

[1012] The AI ​​engine uses a face detection algorithm to identify faces in the image.

[1013] Step 4:

[1014] The AI ​​engine extracts facial features (eye position, nose shape, mouth size, etc.).

[1015] Step 5:

[1016] The server compares the extracted features with face data in an existing database and calculates the degree of match.

[1017] Step 6:

[1018] The server identifies the specific person based on the degree of match and adds them to a photo list of the specific person.

[1019] Emotion recognition processing by emotion engine

[1020] Step 1:

[1021] The user terminal transmits the user's facial expression, tone of voice, text input, etc. in response to the displayed image to the emotion engine.

[1022] Step 2:

[1023] The emotion engine analyzes the user's emotions and generates emotion data (e.g., joy, surprise, sadness, etc.).

[1024] Step 3:

[1025] The emotion data generated by the emotion engine is sent to the server.

[1026] Handling emotion-based display adjustments

[1027] Step 1:

[1028] The server receives the recognized emotion data and selects the next image to display.

[1029] Step 2:

[1030] The server sends the selected image to the terminal.

[1031] Step 3:

[1032] The terminal displays the selected image to the user.

[1033] Handling Notification Functions

[1034] Step 1:

[1035] If the server identifies that a new image has been uploaded to the cloud storage and belongs to a particular person, it generates a notification.

[1036] Step 2:

[1037] The server sends a notification to the user terminal.

[1038] Step 3:

[1039] The device receives the notification and displays the message "New photo added" to the user.

[1040] Photo display and filtering process

[1041] Step 1:

[1042] The user accesses the photo viewing screen and enters search criteria (event name, date, etc.).

[1043] Step 2:

[1044] The terminal sends the search criteria to the server.

[1045] Step 3:

[1046] The server searches for relevant photos from the cloud storage based on the search criteria.

[1047] Step 4:

[1048] The server sends the search results to the user terminal.

[1049] Step 5:

[1050] The terminal receives the search results and displays them to the user.

[1051] Processing learning as we grow

[1052] Step 1:

[1053] The server periodically passes the accumulated image data to the AI ​​engine to prepare for retraining.

[1054] Step 2:

[1055] The server sends the AI ​​engine a new image dataset and instructs it to retrain.

[1056] Step 3:

[1057] The AI ​​engine retrains the facial recognition model using the new dataset.

[1058] Step 4:

[1059] The AI ​​engine returns the retrained model to the server.

[1060] Step 5:

[1061] The server uses the improved model for future facial recognition processing.

[1062] This processing step allows users to efficiently and accurately search for images of specific people and track their developmental progress. The addition of an emotion engine also enables personalized image display that reflects the user's emotions.

[1063] Example 2

[1064] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1065] In modern life, there is a need to efficiently manage large amounts of professionally taken images and easily search for images related to specific people or events. However, conventional systems require users to manually review and categorize images individually, which is time-consuming. Furthermore, they do not adjust image display based on the user's emotions, which hinders the quality of the user experience.

[1066] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for uploading images taken by a professional to cloud storage, means for automatically identifying a person's face from the uploaded image and extracting the facial features, and means for identifying a specific person based on the extracted features. This allows a user to automatically identify images of a specific person and display them in a list. The server also includes means for analyzing the user's emotions regarding the displayed image, means for selecting the next image to display based on the analyzed emotional data, and means for sending a notification when a new image is uploaded. This allows the image display to be adjusted according to the user's emotions, providing a more personalized user experience. Furthermore, the server includes means for continuously analyzing accumulated image data to learn the growth process of a specific person and means for searching and filtering images based on specific events and dates and times, allowing users to quickly and easily search for the images they are looking for.

[1067] "Professional" refers to an individual or organization that has specialized knowledge and skills in a particular field and performs work in that field as a profession.

[1068] "Image" means any representation of visual information in digital or analog form, including photographs, illustrations, graphics, etc.

[1069] "Cloud storage" refers to an online storage service for storing, managing, and backing up data over the Internet.

[1070] "Automatic face identification" refers to the process of detecting a human face in an image and identifying that face using a specific algorithm.

[1071] "Extracting facial features" refers to using a face detection algorithm to obtain the characteristic points and patterns of a person's face (e.g., eye position, nose shape, mouth size, etc.) as digital data.

[1072] "Identifying a specific person" refers to the process of identifying a person based on extracted facial features and comparing them with existing data in a database.

[1073] "Analyzing emotions" refers to the process by which the emotion engine identifies a user's emotional state (e.g., joy, surprise, sadness, etc.) based on data such as the user's facial expressions, tone of voice, and text input.

[1074] "Selecting the next image to display based on emotional data" refers to the process of using analyzed emotional data to automatically select and display an image that suits the user.

[1075] "Send notification" means that the system will notify the user through various methods (e.g., push notification, email, text message, etc.) when new images are uploaded to the cloud storage.

[1076] "Continuously analyzing stored image data" refers to periodically evaluating and analyzing stored image data and continuing to analyze it to identify new characteristics or changes in a particular person.

[1077] "Searching and filtering images based on specific events and dates and times" refers to the process of searching the system for relevant images based on specific criteria specified by the user (e.g., event name, date and time, etc.) and displaying only those images that match those criteria.

[1078] This invention relates to a system for efficiently managing a large number of images taken by professionals and adjusting the display according to the user's emotions. The system is composed of the following elements:

[1079] User terminal

[1080] server

[1081] Cloud Storage

[1082] AI Engine

[1083] Emotion Engine

[1084] First, the images taken by the professionals are uploaded to cloud storage. When the user creates an account using their device, the server validates the input information and stores it in the database, allowing the user to access the system.

[1085] The server then sends the images stored in cloud storage to an AI engine, which uses a face detection algorithm to identify people in the images and extract facial features. The server then compares the extracted features with an existing database to identify specific people. The identified images are added to a list and displayed on the user's device.

[1086] As a user browses images, the emotion engine analyzes the user's reactions (facial expressions, tone of voice, and text input) to identify their emotional state. The analyzed emotion data is sent to the server and used to select the next image to be displayed. For example, if a user expresses positive emotion toward a particular image, the server selects an image with a similar theme and instructs the user's device to display it. Conversely, if a negative emotion is recognized, the server selects an image with a different theme or person.

[1087] Additionally, when new images are uploaded to the cloud storage, the server sends a notification to the user's device, which includes a message that a new image has been added, allowing the user to immediately check the new image.

[1088] Other features of the system include a means to continuously analyze the stored image data to learn the developmental progression of a particular person, and the ability to search and filter images based on specific events or dates and times, allowing users to quickly and easily find the images they are looking for.

[1089] A specific example is shown below.

[1090] 1. A user accesses the application, creates an account, and submits their input information (e.g., "Yamada Taro, first grade elementary school student, 2023").

[1091] 2. The server validates the information, stores it in the database, and sends a notification to the user device that the account was successfully created.

[1092] 3. A professional photographer will upload photos from the April 2023 entrance ceremony to cloud storage.

[1093] 4. The server stores these photos in cloud storage and assigns metadata to each photo.

[1094] 5. The server retrieves the photo from the cloud storage and sends it to the AI ​​engine.

[1095] 6. The AI ​​engine identifies the face, extracts features, and compares them with a database to identify a specific person (e.g., "Taro Yamada").

[1096] 7. The emotion engine analyzes the user's reaction and sends the emotion data to the server.

[1097] 8. The server selects the next image to display based on the emotion data and displays it on the user's device.

[1098] 9. When a new image is uploaded, the server sends a notification and the user device displays it.

[1099] 10. The user sets search filters by event or date and time, and the server provides the corresponding images.

[1100] The following is an example of a prompt:

[1101] "Search for and display photos of Taro Yamada's 2023 entrance ceremony."

[1102] The system optimizes the user experience, allowing parents to easily browse photos from important events, and uses an emotion engine to display personalized images based on the user's emotions.

[1103] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1104] Step 1:

[1105] A user accesses an application and creates an account. The specific actions of this step are that the user enters their name, email address, password, and other required information, and clicks the submit button. The entered data (name, email address, password) is sent to the server. The server receives it and performs validation. If validation is successful, the server saves the data in the database and sends a notification to the user terminal that the account was created successfully. The user terminal displays the message "Account created." The input is the user's information, and the output is a notification that the account was created successfully.

[1106] Step 2:

[1107] The professional photographer uploads the photos he has taken to cloud storage. The specific operations of this step are that the professional photographer selects the image files taken for each event and uploads them to a cloud storage service (e.g., Amazon S3). The uploaded images are assigned metadata such as the event name, date, and location. This information is sent to a server. The server receives this information and stores it in cloud storage. The input is the image files and metadata, and the output is the images stored in cloud storage.

[1108] Step 3:

[1109] The server sends the image stored in cloud storage to the AI ​​engine. The specific operation of this step is that the server retrieves the image file from cloud storage and sends it to the AI ​​engine (e.g., Google Cloud Vision API). The input is the image file retrieved from cloud storage, and the output is the face recognition result by the AI ​​engine. The AI ​​engine uses a face detection algorithm to automatically identify the face in the image and extract facial features (e.g., eye position, nose shape, mouth size, etc.).

[1110] Step 4:

[1111] The server receives the feature data sent from the AI ​​engine and compares it with an existing database. The specific operation of this step is that the server compares the feature data with an existing person database and identifies a specific person based on the degree of match. The input is the feature data obtained from the AI ​​engine, and the output is the result of identifying a specific person. The image of the identified person is added to a list and displayed on the user's device.

[1112] Step 5:

[1113] The user device displays the image and sends the user's reaction to the emotion engine. Specific operations of this step include the user device displaying the identified image and sending the user's facial expression, tone of voice, and text input to the emotion engine (e.g., Microsoft Azure Emotion API). The input is the user's reaction data, and the output is the emotion data identified by the emotion engine.

[1114] Step 6:

[1115] The emotion engine analyzes the user's emotions and sends the results to the server. As a specific operation of this step, the emotion engine analyzes the user's reaction data and identifies the emotional state (e.g., joy, surprise, sadness, etc.). The input is the user's reaction data, and the output is the analyzed emotion data. The server receives this data and selects the next image to display.

[1116] Step 7:

[1117] The server selects the next image to display based on the emotion data and instructs the user terminal. As a specific operation of this step, the server generates a list of images to display next based on the analyzed emotion data. For example, if the user expresses positive emotion toward a particular image, the server selects images with a similar theme. The input is the analyzed emotion data, and the output is a list of images to display next. The user terminal displays the images according to this instruction.

[1118] Step 8:

[1119] The server sends a notification when a new image is uploaded. The specific operation of this step is that the server monitors the cloud storage and sends a notification to the user device when a new image is uploaded. The input is the newly uploaded image, and the output is the notification to the user device. The user device displays a notification saying "A new photo has been added."

[1120] Step 9:

[1121] The user sets a search filter based on an event or date and time, and the server provides the relevant images. In this step, the user enters search criteria based on a specific event or date and time within the application and presses the search button. The input is the search filter set by the user, and the output is a list of relevant images. The server searches for relevant images based on this request and sends the results to the user's device. The user's device displays the search results.

[1122] Example prompt sentence:

[1123] "Search and view photos from the 2023 Commencement Ceremony."

[1124] (Application example 2)

[1125] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1126] Existing image management systems are required to efficiently classify images taken by professionals and provide a means to easily search and display the images they need. However, the lack of a function to dynamically change the display content based on user sentiment limits the ability to improve the user experience. Furthermore, it is often difficult for end users to effectively search and filter images based on specific events or dates and times. A new system that solves this problem is needed.

[1127] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading images taken by a professional to cloud storage, means for automatically identifying people's faces from the uploaded images and extracting their facial features, means for identifying specific people based on the extracted features, means for displaying a list of images of the identified specific people, means for sending a notification when a new image is uploaded, means for recognizing a user's facial expression in real time and identifying their emotion, means for adjusting the next image to be displayed based on the identified emotion, means for continuously analyzing accumulated image data to learn the growth process of a specific person, and means for searching and filtering images based on specific events and dates and times. This allows the server to provide display content optimized based on the user's emotions, making it easy for end users to search and filter for the images they need.

[1128] "Professionally captured images" are high-quality images captured using specialized techniques and equipment.

[1129] "Cloud storage" refers to external data storage services accessible over the Internet that allow users to store and manage data remotely.

[1130] "Means for automatically identifying a person's face" is a technology that detects a person's face in an image and automatically recognizes the position and outline of that face.

[1131] "Means for extracting facial features" refers to a technology that analyzes features such as the eyes, nose, and mouth from a person's face and extracts them as data.

[1132] The "means for identifying a specific person" is a technology that recognizes a specific person by comparing the extracted facial features with facial data registered in a database.

[1133] The "means for displaying images in a list" is a technology that visually organizes images related to an identified person and displays them all together on the screen.

[1134] "Means for sending notifications" refers to technology that sends alerts or messages to user terminals when new information is added.

[1135] "Means for recognizing a user's facial expression in real time" refers to a technology that captures the user's face with a camera, instantly analyzes their facial expression, and determines their emotional state.

[1136] The "means for identifying emotions" is a technology that determines the user's emotions (for example, joy, surprise, sadness, etc.) based on analyzed facial expression data.

[1137] The "means for adjusting the next image to be displayed" is a technique for changing the order or content of images to be presented to the user based on the recognized emotion.

[1138] "Means for learning growth processes" refers to technology that analyzes image data accumulated over time to learn about the growth and changes of a specific person.

[1139] "Means for searching and filtering images" refers to a technique for specifying images based on specific conditions (for example, event name or date and time) and extracting images that meet the purpose.

[1140] The system of the present invention efficiently manages images taken by professionals and improves the user experience by dynamically displaying images according to the user's emotions. This system mainly consists of the following elements.

[1141] Server, user terminal, cloud storage, AI engine, and emotion engine.

[1142] 1. User Registration

[1143] When a user first accesses the application, the user creates an account on the user's device. During this process, the user enters basic information. The server receives this information, validates it, and then saves it in the database to create the account.

[1144] 2. Upload a photo

[1145] Professionally taken images are uploaded to cloud storage. The server receives these images and stores them in cloud storage along with metadata (e.g., event name, date, location).

[1146] 3. AI-powered facial recognition and classification

[1147] The server passes the images stored in cloud storage to an AI engine, which uses a face detection algorithm to identify faces in the photos, identifies specific people based on the extracted facial features, and stores the results in a database.

[1148] 4. Emotion Recognition by Emotion Engine

[1149] The user device analyzes the user's reaction to the displayed images in real time through an emotion engine, which analyzes the user's facial expressions and tone of voice to identify emotions.

[1150] 5. Adjusting display based on emotions

[1151] The server then adjusts the next image to be displayed based on the recognized emotion: if a positive emotion is detected, images with related themes are prioritized; if a negative emotion is detected, images with different themes or people are displayed instead.

[1152] 6. Notification function

[1153] When a new image is uploaded to cloud storage and is recognized by the AI ​​engine as an image of a specific person, the server sends a notification to the user's device, allowing the user to immediately view the new image.

[1154] 7. Viewing and filtering photos

[1155] Users can access the photo browsing screen within the application and filter photos based on events or dates. The server generates a list of photos based on the user's request and sends it to the user's device, allowing the user to visually view the photos.

[1156] 8. Learning the process of growth

[1157] The server continuously trains the AI ​​engine based on the stored image data, tracking the development of specific individuals and improving the accuracy of identification with each new data point.

[1158] Hardware and software used

[1159] The following hardware and software are specifically used to implement this system:

[1160] Hardware:

[1161] 1. Smart glasses (e.g., Google Glass)

[1162] 2. Camera

[1163] software:

[1164] 1. OpenCV (for face recognition)

[1165] 2. EmotionRecognizer (emotion recognition library)

[1166] 3. ProductRecommender (product recommendation engine)

[1167] Specific examples

[1168] As a user walks through a virtual store wearing smart glasses, the camera in the smart glasses captures the user's face and analyzes their facial expressions in real time. For example, if a user smiles after looking at a product, the emotion engine will identify that positive emotion and instantly recommend and display relevant products. In this way, it is possible to suggest products tailored to the user's emotional state.

[1169] Prompt example (text format)

[1170] Design a program that allows smart glasses to recognize the user's facial expressions in real time as they walk through a virtual store and display products according to their emotions. Include the following elements:

[1171] 1. Detect faces from the camera stream.

[1172] 2. Uses facial recognition and emotion identification algorithms.

[1173] 3. If positive sentiment is detected, product recommendations are displayed.

[1174] 4. Specify the hardware and software you will be using.

[1175] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1176] Step 1:

[1177] When a user walks through the virtual store, the user device (smart glasses) takes a picture of the user's face in real time using the built-in camera. As input, it takes the video frames captured by the camera and passes them to the next processing step. As output, it obtains the captured video frames.

[1178] Step 2:

[1179] The server receives the video frame and runs a face detection algorithm using OpenCV. The pixel data of the video frame is passed as input, and the area containing the face is identified. The data is then processed to extract the face position and bounding box. The output is a video frame containing face position information.

[1180] Step 3:

[1181] Based on the identified facial area, the server uses EmotionRecognizer to analyze facial expressions and identify emotions. The input is the position information of the face and the pixel data of that area. The data is then processed by analyzing facial features (e.g., eye shape and mouth open / closed degree) and classifying the emotion (happiness, surprise, etc.). The output is the identified emotion data.

[1182] Step 4:

[1183] The server receives the emotion data and adjusts the next image to be displayed based on the identified emotion. The emotion data and metadata of the products in the virtual store are passed as input. The data is then processed to select products that match the emotion data and generate a list of highly relevant products. The output is the selected product list.

[1184] Step 5:

[1185] Based on the product list, the server sends the product information to be displayed next to the user's terminal. The selected product list is passed as input. Data processing involves packaging the product information in a format appropriate for the user's terminal. The packaged product information is obtained as output.

[1186] Step 6:

[1187] The user terminal displays the product on the smart glasses display based on the received product information. The product information sent from the server is passed as input. Specific operations include displaying product images and explanatory text on the display, visually presenting the product to the user. The output is a product display that the user can visually confirm.

[1188] Through the above processing steps, product suggestions based on the user's emotions are realized in real time, and a system that enhances the user experience is constructed.

[1189] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1190] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1191] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1192] [Third embodiment]

[1193] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1194] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1195] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1196] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1197] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1198] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1199] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1200] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1201] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1202] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1203] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1204] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1205] ---

[1206] This invention is a system for automatically classifying images and tracking the growth process of specific individuals. This system uploads professionally taken images to cloud storage, uses an AI engine to identify faces, extracts their features, and displays a list of images of the identified specific individuals.

[1207] System Configuration

[1208] The system mainly consists of the following components:

[1209] User terminal

[1210] server

[1211] Cloud Storage

[1212] AI Engine

[1213] User Registration

[1214] When a user first accesses the application, the user creates an account on the user's device. During this process, the user enters basic information such as an email address, password, child's name, and school year. The server receives this information, validates it, and stores it in the database. This creates a new account and sends the user a notification that the account was created successfully.

[1215] Upload a photo

[1216] Professionally taken images are uploaded to cloud storage. The professional photographer compiles a large number of photos taken at events (e.g., athletic meets, entrance ceremonies, etc.) and adds metadata (event name, date, location, etc.) to each photo. The server receives these images and stores them in cloud storage along with the metadata.

[1217] AI-powered facial recognition and classification

[1218] The server passes the images stored in cloud storage to the AI ​​engine, which uses a face detection algorithm to identify faces in the photos. During this process, facial features (such as eye position, nose shape, and mouth size) are extracted. The server then compares these features with facial data in an existing database and identifies specific people based on the degree of match. The identified images are added to a list of images that are determined to contain specific people.

[1219] Notification function

[1220] When a new image is uploaded to the cloud storage and is recognized by the AI ​​as an image of a specific person, the server sends a notification to the user's device. This notification includes information that a new image has been added. The user's device receives this notification and notifies the user via a pop-up message or other means.

[1221] Viewing and filtering photos

[1222] Users can access the photo browsing screen within the application and set filtering criteria to search for photos based on specific events or dates. The server generates a list of photos based on the user's request and sends it to the user's device, which then visually displays the received photos.

[1223] Learning the process of growth

[1224] The server continuously trains the AI ​​engine based on the accumulated image data. This process tracks the development of specific individuals and improves the accuracy of identification with each new data set. The new data set is used to retrain the facial recognition model, optimizing the overall performance of the system.

[1225] Specific examples

[1226] User Registration

[1227] 1. The user accesses the application and enters the required information on the account creation screen (e.g., Yamada Taro, first grade elementary school student in 2023).

[1228] 2. The server retrieves this information, validates it, and if successful, saves it in the database and creates an account.

[1229] 3. The user device will display a message indicating that the account was created successfully.

[1230] Upload a photo

[1231] 1. A professional photographer will compile photos from the April 2023 entrance ceremony and upload them to cloud storage.

[1232] 2. The server stores these photos in cloud storage and assigns metadata to each photo.

[1233] AI-powered facial recognition and classification

[1234] 1. The server passes the uploaded photo to the AI ​​engine and performs facial recognition.

[1235] 2. The AI ​​engine identifies faces in each photo and extracts features.

[1236] 3. The server identifies a specific person from the database based on the features and adds the image to a list based on the degree of match (e.g., identifying it as a photo of Taro Yamada).

[1237] Notification function

[1238] 1. When the server identifies a new photo as belonging to a specific person, it sends a notification to the user's device.

[1239] 2. The user device displays the message "New photo added."

[1240] Viewing and filtering photos

[1241] 1. The user accesses the photo viewing screen, sets a filter such as "Events: Entrance Ceremony 2023," and performs a search.

[1242] 2. The server searches for the relevant photos and sends the results to the user's device.

[1243] 3. The user terminal displays the search results to the user.

[1244] This allows users to efficiently search for photos based on specific events or dates. Furthermore, as the AI ​​engine continues to learn, its identification accuracy improves, making it easier to track the growth process of a specific person. This system prevents parents from overlooking photos of important events and makes it easier to search for commemorative photos.

[1245] The processing flow will be explained below.

[1246] Handling user registration

[1247] Step 1:

[1248] The user accesses the application and enters the required information (email address, password, child's name, grade, etc.) on the account creation screen.

[1249] Step 2:

[1250] The terminal transmits the input information to the server.

[1251] Step 3:

[1252] The server validates the information received and checks the format and content of the input information.

[1253] Step 4:

[1254] The server saves the validated information to the database and creates the new account.

[1255] Step 5:

[1256] The server notifies the device that the account creation was successful.

[1257] Step 6:

[1258] The device displays a message to the user confirming successful account creation.

[1259] Handling photo uploads

[1260] Step 1:

[1261] A professional photographer takes images from the event and prepares them for upload to cloud storage, where they are tagged with metadata (event name, date, location, etc.).

[1262] Step 2:

[1263] The device sends the image with the metadata to the server.

[1264] Step 3:

[1265] The server stores the received image data in cloud storage.

[1266] AI-powered facial recognition and classification

[1267] Step 1:

[1268] The server prepares the images stored in cloud storage to be passed to the AI ​​engine.

[1269] Step 2:

[1270] The server sends the image data to the AI ​​engine and instructs it to begin facial recognition processing.

[1271] Step 3:

[1272] The AI ​​engine uses a face detection algorithm to identify faces in the image.

[1273] Step 4:

[1274] The AI ​​engine extracts facial features (eye position, nose shape, mouth size, etc.).

[1275] Step 5:

[1276] The server compares the extracted features with face data in an existing database and calculates the degree of match.

[1277] Step 6:

[1278] The server identifies the specific person based on the degree of match and adds them to a list of photos of the specific person.

[1279] Handling Notification Functions

[1280] Step 1:

[1281] The server generates a notification when a new image is identified as being of a particular person.

[1282] Step 2:

[1283] The server sends a notification to the user terminal.

[1284] Step 3:

[1285] The device receives the notification and displays the message "New photo added" to the user.

[1286] Photo display and filtering process

[1287] Step 1:

[1288] The user accesses the photo viewing screen and enters search criteria (e.g., event name, date, etc.).

[1289] Step 2:

[1290] The terminal transmits the entered search conditions to the server.

[1291] Step 3:

[1292] Based on the search criteria received by the server, the server searches for corresponding photos from the cloud storage.

[1293] Step 4:

[1294] The server sends the search results to the user terminal.

[1295] Step 5:

[1296] The device receives the search results, organizes them visually, and displays them to the user.

[1297] Processing learning as we grow

[1298] Step 1:

[1299] The server periodically passes the accumulated image data to the AI ​​engine to prepare for retraining.

[1300] Step 2:

[1301] The server sends the AI ​​engine a new image dataset and instructs it to retrain.

[1302] Step 3:

[1303] The AI ​​engine retrains the facial recognition model using the new dataset.

[1304] Step 4:

[1305] The AI ​​engine returns the retrained model to the server.

[1306] Step 5:

[1307] The server uses the improved model for future facial recognition processing.

[1308] These are the specific processing steps of this system, allowing users to quickly and accurately find images of specific people and efficiently track their growth process.

[1309] Example 1

[1310] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1311] Modern digital photo management requires efficient classification of large amounts of photo data and rapid retrieval of photos related to specific people or events. However, manual classification and retrieval is laborious and time-consuming. Improving identification accuracy is particularly important when tracking the growth of specific people. Conventional technologies have not adequately addressed these challenges, and further improvements in the user experience are needed.

[1312] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1313] In this invention, the server includes a means for an end user to input information and create an account, a means for uploading images taken by a professional to network storage, a means for automatically identifying people's faces from the uploaded images and extracting facial features, a means for sending a notification to a user terminal when a new image of a specific person is uploaded, and a means for displaying or filtering the identified images of the specific person based on search conditions. This makes it possible to quickly and accurately search for images related to a specific person or event from a large amount of photo data, and also to track their growth.

[1314] An "end user" is the final user of the system, who performs operations such as uploading, searching, and viewing photos.

[1315] An "expert" is someone who has the skills and knowledge to take photographs and can provide high-quality images.

[1316] "Network storage" is an online storage device accessible via the Internet, and is a system for saving and managing data.

[1317] "Automatic face identification" is the process of using image analysis technology to identify the faces of people present in a photograph.

[1318] "Features" refer to the unique attributes and shape information required to identify a face, and by extracting these, individual people can be recognized.

[1319] A "specific person" is an individual identified based on facial features registered in a database.

[1320] A "user terminal" is a device that connects to the system and operates it, and includes smartphones, tablets, PCs, etc.

[1321] A "notification" is a message from the system to inform the user of new information or events.

[1322] "Search conditions" are filters and keywords that a user sets when searching for a specific photo, and include the event name, date and time, etc.

[1323] "Growth tracking" is the process of analyzing accumulated image data to track the changes in a particular person over the years.

[1324] The present invention provides a system for automatically classifying images and tracking the growth process of a specific person. The system includes a means for an end user to input information and create an account, a means for uploading images taken by a professional to a network storage, a means for automatically identifying faces of people from the uploaded images and extracting facial features, a means for sending a notification to a user terminal when a new image of the specific person is uploaded, and a means for displaying or filtering the identified images of the specific person based on search criteria.

[1325] Hardware and software used

[1326] The system mainly consists of the following components:

[1327] User devices: smartphones, tablets, computers, etc.

[1328] Server: High-performance computing server

[1329] Network storage: Cloud storage services (e.g., Amazon S3 or Google Cloud Storage)

[1330] AI Engine: Machine learning models (e.g., TensorFlow or OpenCV) for face recognition and feature extraction

[1331] Example of user registration

[1332] A user accesses an application and creates an account. For example, the user enters the email address "user@example.com", the password "securepassword", the child's name "Child A", and the grade "First grade of elementary school". The device sends this information to the server, which validates the information. The server then saves the information in the database and a new account is created. The user's device displays a message indicating that the account was created successfully.

[1333] Example of photo upload

[1334] An expert (e.g., a professional photographer) takes photos of an event (e.g., an entrance ceremony) and uploads them to network storage. When uploading, the photos are given metadata such as the event name "Entrance Ceremony 2023," the date "April 1, 2023," and the location "Elementary School A." The server receives these photos and metadata and stores them in network storage.

[1335] Specific examples of AI facial recognition and classification

[1336] The server periodically checks the network storage and detects new photos. It passes the new photos to the AI ​​engine for facial recognition processing. The AI ​​engine uses a face detection algorithm to identify faces in the photos and extract features such as eye position, nose shape, and mouth size. The server compares these features with facial data in an existing database and identifies a specific person (e.g., Child A) based on the degree of match. The server adds the identification results to a list of photos of specific people.

[1337] Examples of notification features

[1338] When the server identifies a new photo of a specific person (e.g., Child A), it sends a notification to the user device saying, "A new photo has been added." The user device receives this notification and notifies the user via a pop-up message or similar.

[1339] Examples of photo display and filtering

[1340] The user accesses the photo viewing screen within the application and sets filtering conditions for a specific event, such as "Entrance Ceremony 2023," or a date, such as "April 1, 2023." The server searches for matching photos based on the user's request and generates results. The user's device displays a list of search results.

[1341] Examples of learning in the process of growth

[1342] The server continuously trains the AI ​​engine based on the accumulated image data. For example, the AI ​​engine retrains based on past photos of Child A, and improves its recognition accuracy by uploading new photos.

[1343] Example prompts for generative AI models

[1344] "Please explain the user registration process in detail, from when the user enters the required information, through when the server validates and saves the information to the database, and notifies the user that the account was successfully created."

[1345] "Please explain in detail the process of uploading photos. Please explain in detail the process from an expert uploading photos to network storage to the server storing them with metadata."

[1346] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1347] Step 1: User Registration

[1348] 1. The user accesses the application and enters the required information (email address, password, child's name, grade, etc.) on the account creation screen.

[1349] Input: Email address "user@example.com", password "securepassword", child's name "Child A", grade "First grade of elementary school"

[1350] Specific operation: The user enters information using the input form and presses the "Register" button.

[1351] 2. The device sends the entered information to the server and requests account creation.

[1352] Output: Data packet containing user information

[1353] Specific operation: The terminal generates a data packet and sends it to the server.

[1354] 3. Validate the information received by the server (check the format of the email address, verify the strength of the password, etc.).

[1355] Input: Data packet containing user information

[1356] Output: Validation result (pass or fail)

[1357] What happens: The server checks the format of the email address and evaluates the strength of the password. If validation is successful, it proceeds to the next step.

[1358] 4. If the server validates successfully, it saves the information in the database and creates a new account.

[1359] Input: User information that has been successfully validated

[1360] Output: Account information stored in the database

[1361] Specific operation: The server opens a database connection and registers the user information as a new record.

[1362] 5. Send a notification to the device that the account was created successfully.

[1363] Input: Account creation success status

[1364] Output: Account creation successful message

[1365] Specific operation: The server generates a success message and sends it to the terminal, which displays the message.

[1366] Step 2: Upload a photo

[1367] 1. Professionals take photos at the event and upload them to network storage.

[1368] Input: Photo file

[1369] Output: Upload request to network storage

[1370] What happens: The expert uses the upload interface to select a photo file and begin uploading.

[1371] 2. An expert assigns metadata to each photo (e.g., event name "Entrance Ceremony 2023", date "April 1, 2023", location "Elementary School A").

[1372] Input: Photo file, metadata

[1373] Output: Photo files with metadata

[1374] Specific operation: The expert enters the required information into the metadata input form and attaches the metadata to the photo.

[1375] 3. The server receives the uploaded photos and metadata and stores them in network storage.

[1376] Input: Photo files with metadata

[1377] Output: Photos stored on network storage

[1378] Specific operation: The server connects to network storage and stores photos and metadata.

[1379] Step 3: AI-powered facial recognition and classification

[1380] 1. The server periodically checks the cloud storage for new photos.

[1381] Input: Photo data from cloud storage

[1382] Output: A list of new photos

[1383] Specific operation: The server sets a timer and periodically accesses the cloud storage to check for new photo files.

[1384] 2. The server passes the new photo to the AI ​​engine for facial recognition processing.

[1385] Input: New photo file

[1386] Output: Face recognition results

[1387] Specific operation: The server passes the photo file to the AI ​​engine, which runs the face detection algorithm.

[1388] 3. The AI ​​engine uses a face detection algorithm to identify faces in the photo and extract their features.

[1389] Input: Photo file

[1390] Output: Facial feature data

[1391] Specific operation: The AI ​​engine detects the face and extracts features such as eye position, nose shape, and mouth size.

[1392] 4. The server compares the features with facial data in an existing database and identifies the person based on the degree of match.

[1393] Input: Facial feature data

[1394] Output: Identification result of a specific person

[1395] Specific operation: The server compares the feature data with face data in the database, calculates the degree of match, and identifies the person.

[1396] 5. The server adds the new photo to the photo list of the identified person.

[1397] Input: Identification result of a specific person

[1398] Output: Updated list of photos of specific people

[1399] Specific operation: The server adds the photo to the list of specific people and updates the list.

[1400] Step 4: Notifications

[1401] 1. The server sends a notification to the user device when a new photo of a specific person is identified.

[1402] Input: Identification results for new photos of a specific person

[1403] Output: Notification message

[1404] Specific operation: The server generates a notification message and sends it to the user terminal.

[1405] 2. The user device receives the notification and notifies the user via a pop-up message or similar.

[1406] Input: Notification message

[1407] Output: Display a popup message

[1408] Specific operation: The user terminal receives the notification message and displays a pop-up message.

[1409] Step 5: View and filter photos

[1410] 1. The user accesses the photo viewing screen within the application and sets filtering criteria for specific events or dates.

[1411] Input: Filtering criteria such as event name and date

[1412] Output: Search request based on filtering criteria

[1413] Specific operation: The user enters conditions into the filtering form and presses the search button.

[1414] 2. The server searches for relevant photos based on the user's request and generates the results.

[1415] Input: Filtering criteria

[1416] Output: A list of photos as search results

[1417] Specific operation: The server searches the photos in the database and generates a list of photos that match the criteria.

[1418] 3. The user device receives the search results and displays a list of photos.

[1419] Input: Photo list as search results

[1420] Output: List of photos

[1421] Specific operation: The user terminal loads the photo list onto the display screen and displays a visual list.

[1422] Step 6: Learning the process of growth

[1423] 1. The server continuously trains the AI ​​engine based on the accumulated image data.

[1424] Input: Stored image data

[1425] Output: Updated AI model

[1426] Specific operation: The server supplies image data to the AI ​​engine and causes it to re-learn.

[1427] 2. The server uses the new dataset to retrain the facial recognition model, improving the system's overall recognition accuracy.

[1428] Input: New dataset, existing AI model

[1429] Output: Retrained AI model

[1430] Specific operation: The server retrains the AI ​​model using a new dataset to improve its recognition accuracy.

[1431] (Application example 1)

[1432] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1433] Providing personalized services to modern consumers is extremely important in increasing customer satisfaction. However, it is difficult to quickly and accurately grasp customer attributes in physical stores and provide services that are appropriate for the customer on the spot. As a result, customer needs cannot be adequately met and store profits are not maximized. It is also difficult to provide a consistent customer experience to multiple customers who visit a store. To solve these issues, a means is needed to use advanced technology to provide personalized services based on customer attributes.

[1434] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1435] In this invention, the server includes means for uploading images taken by a professional to cloud storage, means for automatically identifying people's faces from the uploaded images and extracting their facial features, means for identifying specific people based on the extracted features, means for displaying a list of images of the identified specific people, means for sending a notification when a new image is uploaded, means for identifying the faces of customers visiting the store and specifying the customer's attributes, and means for providing personalized services based on the specified customer attributes. This makes it possible to easily determine customer attributes even in physical stores and provide optimal services to each customer.

[1436] A "professional" is someone who has specialized skills and knowledge and makes a living from them.

[1437] "Cloud storage" refers to remote servers that provide services for storing and managing data over the Internet.

[1438] "Means for identifying human faces" refers to methods and devices that use cameras and image analysis software to identify and recognize human faces in images.

[1439] "Means for extracting facial features" refers to a method of analyzing and acquiring features such as facial shape, pattern, and location information using a facial recognition algorithm.

[1440] The "means for identifying a specific person" refers to a method and apparatus that uses the extracted facial features to match existing facial data in a database and identify matching people.

[1441] "Means for displaying in a list" refers to a method for displaying images of identified specific persons together on a display or screen in a visually easy-to-understand format.

[1442] "Means of sending notifications" refers to the methods of sending messages or alerts to users when new images are uploaded.

[1443] "Means for identifying the faces of customers visiting a store" refers to a method and device for photographing the faces of customers using a camera or other device installed in a physical store and recognizing the faces of customers from the images.

[1444] "Means for identifying customer attributes" refers to a method for analyzing and identifying a customer's age, gender, and other attribute information based on recognized facial features.

[1445] The "means for providing personalized services" refers to a method and apparatus for providing products and services optimized for a customer based on the identified customer's attribute information.

[1446] This invention is a system that identifies customers' faces in brick-and-mortar stores and provides personalized services based on their attributes. The system is mainly composed of a server, cloud storage, an AI engine, and a user's device. The specific configuration and operation of this system are described below.

[1447] User Registration

[1448] When a user first accesses the system through a smartphone application, they create an account. They enter their email address, password, and other basic information, and the server receives, validates, and stores this information in a database.

[1449] Upload a photo

[1450] It provides a way for professionally taken images to be uploaded to cloud storage, where the professional adds metadata such as the event name, date, and location to each photo, and the server receives and stores the metadata in the cloud storage.

[1451] AI-powered facial recognition and classification

[1452] The server passes the images stored in cloud storage to an AI engine for facial identification, which uses a facial recognition algorithm to extract facial features and match them with an existing database to identify a specific person.

[1453] Notification function

[1454] When a new image is uploaded to the cloud storage and identified by the AI ​​engine as belonging to a specific person, the server sends a notification to the user's device, which includes information that a new image has been added.

[1455] Viewing and filtering photos

[1456] Users can access the photo browsing screen in the application and search for photos based on specific events or dates and times. The server searches for the relevant photos based on the user's request and sends them to the user's device.

[1457] Identifying the faces of customers visiting a store

[1458] Using cameras and smartphones installed in physical stores, the server photographs and recognizes the faces of customers visiting the store, extracts their facial features, and uses that information to identify their attributes (age, gender, etc.).

[1459] Providing personalized service

[1460] The server customizes the store's services based on the identified customer attributes, for example, by providing recommended products and specific campaign information.

[1461] Specific examples

[1462] 1. User Registration:

[1463] A user accesses the application and enters basic information.

[1464] The server receives the information and stores it in a database.

[1465] 2. Upload a photo:

[1466] Professionally captured images are uploaded to cloud storage.

[1467] The server receives the images and stores them along with the metadata.

[1468] 3. Identifying the faces of customers visiting your store:

[1469] Cameras installed at the entrances of physical stores capture customers' faces.

[1470] The server analyzes facial features and identifies customer attributes.

[1471] 4. Service Provision:

[1472] The server provides the optimal service to the customer based on the identified customer attribute information.

[1473] Prompt Sentence Examples

[1474] "How can we provide specific services that are best suited to store customers based on facial recognition results? Please provide the following information:

[1475] Facial image

[1476] Customer attributes (age, gender)

[1477] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1478] Step 1:

[1479] A user creates an account through a smartphone application. The user enters their email address, password, and other basic information. The information is sent to the server, where it is validated and stored in a database.

[1480] Input: User's email address, password, basic information

[1481] Data processing: information validation

[1482] Output: Account information is saved in the database

[1483] Step 2:

[1484] Professionally taken images are uploaded to cloud storage, and metadata such as the event name, date, and location are added to the images. The server receives these and stores them in cloud storage.

[1485] Input: Professionally captured images and metadata

[1486] Data Computing: Saving images and metadata to cloud storage

[1487] Output: Images and metadata are saved to cloud storage

[1488] Step 3:

[1489] The server passes the images stored in cloud storage to an AI engine that identifies faces, which uses a facial recognition algorithm to extract facial features and match them with existing databases.

[1490] Input: Images stored in cloud storage

[1491] Data Computing: Facial feature extraction and database matching

[1492] Output: Information about the specific person identified

[1493] Step 4:

[1494] When a new image is uploaded and the AI ​​engine identifies a specific person, the server sends a notification to the user's device, which includes information that a new image has been added.

[1495] Input: Information that identifies a specific person

[1496] Data processing: Notification message generation

[1497] Output: Notification to user terminal

[1498] Step 5:

[1499] The user accesses the photo viewing screen within the application and searches for photos based on a specific event or date and time. The server searches for the corresponding photos based on the user's request and sends them to the user's device.

[1500] Input: User request (event name and date and time)

[1501] Data calculation: Search for relevant photos

[1502] Output: Send search results to the user's terminal

[1503] Step 6:

[1504] A camera installed at the entrance of a physical store captures the faces of customers. The server receives the captured images, analyzes their facial features, and identifies the customer's attributes (age, gender).

[1505] Input: An image of the customer captured by a camera

[1506] Data Computing: Facial feature analysis and attribute identification

[1507] Output: Customer attribute information

[1508] Step 7:

[1509] The server customizes the store's services based on the identified customer attribute information, for example, by recommending specific products or providing specific campaign information.

[1510] Input: Customer attribute information

[1511] Data processing: Creating customized service content

[1512] Output: personalized service presentation

[1513] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1514] ---

[1515] The system uploads professionally taken images to cloud storage, uses an AI engine to identify faces, extracts their features to recognize specific people, and uses an additional emotion engine to identify the user's emotions, thereby personalizing the images displayed. The system consists of the following components:

[1516] User terminal

[1517] server

[1518] Cloud Storage

[1519] AI Engine

[1520] Emotion Engine

[1521] User Registration

[1522] When a user first accesses the application, the user creates an account on the user's device. During this process, the user enters basic information such as an email address, password, child's name, and school year. The server receives this information, validates it, and stores it in the database. This creates a new account and sends the user a notification that the account was created successfully.

[1523] Upload a photo

[1524] Professionally taken images are uploaded to cloud storage. The professional photographer compiles a large number of photos taken at the event and adds metadata (event name, date, location, etc.) to each photo. The server receives these images and stores them in cloud storage along with the metadata.

[1525] AI-powered facial recognition and classification

[1526] The server passes the images stored in cloud storage to the AI ​​engine, which uses a face detection algorithm to identify faces in the photos. During this process, facial features (such as eye position, nose shape, and mouth size) are extracted. The server then compares these features with facial data in an existing database and identifies specific people based on the degree of match. The identified images are added to a list of images that are determined to contain specific people.

[1527] Emotion recognition by emotion engine

[1528] The user device analyzes the user's reaction to the displayed image through an emotion engine, which analyzes the user's facial expressions, tone of voice, text input, etc. to identify emotions (happiness, surprise, sadness, etc.).

[1529] Emotion-based display adjustment

[1530] Based on the perceived emotion, the device will adjust the next image displayed: for example, if the user expresses positive emotion toward a particular image, it will continue to display images with a similar theme, or if negative emotion toward a particular image is detected, it will switch to images with a different theme or person.

[1531] Notification function

[1532] When a new image is uploaded to the cloud storage and is recognized by the AI ​​as an image of a specific person, the server sends a notification to the user's device that a new image has been added, allowing the user to immediately view the new image.

[1533] Viewing and filtering photos

[1534] Users can access the photo browsing screen within the application and set filtering criteria to search for photos based on specific events or dates. The server generates a list of photos based on the user's request and sends it to the user's device, which then visually displays the received photos.

[1535] Learning the process of growth

[1536] The server continuously trains the AI ​​engine based on the accumulated image data. This process tracks the development of specific individuals and improves the accuracy of identification with each new data set. The new data set is used to retrain the facial recognition model, optimizing the overall performance of the system.

[1537] Specific examples

[1538] User Registration

[1539] 1. The user accesses the application and enters the required information on the account creation screen (e.g., Yamada Taro, first grade elementary school student in 2023).

[1540] 2. The server retrieves this information, validates it, and if successful, saves it in the database and creates an account.

[1541] 3. The user device will display a message indicating that the account was created successfully.

[1542] Upload a photo

[1543] 1. A professional photographer will compile photos from the April 2023 entrance ceremony and upload them to cloud storage.

[1544] 2. The server stores these photos in cloud storage and assigns metadata to each photo.

[1545] AI-powered facial recognition and classification

[1546] 1. The server passes the uploaded photo to the AI ​​engine and performs facial recognition.

[1547] 2. The AI ​​engine identifies faces in each photo and extracts features.

[1548] 3. The server identifies a specific person from the database based on the features and adds the image to a list based on the degree of match (e.g., identifying it as a photo of Taro Yamada).

[1549] Emotion recognition by emotion engine

[1550] 1. The emotion engine analyzes the user's reaction to the displayed image (facial expression, tone of voice, text input).

[1551] 2. The emotion engine identifies the user's emotion and sends the data to the server.

[1552] Emotion-based display adjustment

[1553] 1. The server selects the next image to display based on the recognized emotion.

[1554] 2. If a user expresses positive feelings about a particular image, the device is instructed to display images with a similar theme.

[1555] 3. If the user expresses negative emotions, instruct the device to switch to an image of a different subject or person.

[1556] Notification function

[1557] 1. When the server identifies a new photo as belonging to a specific person, it sends a notification to the user's device.

[1558] 2. The user device displays the message "New photo added."

[1559] Viewing and filtering photos

[1560] 1. The user accesses the photo viewing screen, sets a filter such as "Events: Entrance Ceremony 2023," and performs a search.

[1561] 2. The server searches for the relevant photos and sends the results to the user's device.

[1562] 3. The user terminal displays the search results to the user.

[1563] This allows users to efficiently search for photos based on specific events or dates. Furthermore, as the AI ​​engine continues to learn, its identification accuracy improves, making it easier to track the growth process of a specific person. The addition of an emotion engine further improves the user experience by identifying the user's emotions and personalizing the images displayed. This system prevents parents from overlooking photos of important events and makes it easier to search for and display commemorative photos.

[1564] The processing flow will be explained below.

[1565] Handling user registration

[1566] Step 1:

[1567] The user accesses the application and enters the required information (email address, password, child's name, grade, etc.) on the account creation screen.

[1568] Step 2:

[1569] The terminal transmits the input information to the server.

[1570] Step 3:

[1571] The server validates the information received and checks the format and content of the input information.

[1572] Step 4:

[1573] The server saves the validated information to the database and creates the new account.

[1574] Step 5:

[1575] The server notifies the device that the account creation was successful.

[1576] Step 6:

[1577] The device displays a message to the user confirming successful account creation.

[1578] Handling photo uploads

[1579] Step 1:

[1580] A professional photographer prepares images taken at an event (e.g., athletic meet, entrance ceremony, etc.) for uploading to cloud storage.

[1581] Step 2:

[1582] A professional photographer adds metadata (event name, date, location, etc.) to the images.

[1583] Step 3:

[1584] The device sends the image with the metadata to the server.

[1585] Step 4:

[1586] The server stores the received image data in cloud storage.

[1587] AI-powered facial recognition and classification

[1588] Step 1:

[1589] The server prepares the images stored in cloud storage to be passed to the AI ​​engine.

[1590] Step 2:

[1591] The server sends the image data to the AI ​​engine and instructs it to begin facial recognition processing.

[1592] Step 3:

[1593] The AI ​​engine uses a face detection algorithm to identify faces in the image.

[1594] Step 4:

[1595] The AI ​​engine extracts facial features (eye position, nose shape, mouth size, etc.).

[1596] Step 5:

[1597] The server compares the extracted features with face data in an existing database and calculates the degree of match.

[1598] Step 6:

[1599] The server identifies the specific person based on the degree of match and adds them to a photo list of the specific person.

[1600] Emotion recognition processing by emotion engine

[1601] Step 1:

[1602] The user terminal transmits the user's facial expression, tone of voice, text input, etc. in response to the displayed image to the emotion engine.

[1603] Step 2:

[1604] The emotion engine analyzes the user's emotions and generates emotion data (e.g., joy, surprise, sadness, etc.).

[1605] Step 3:

[1606] The emotion data generated by the emotion engine is sent to the server.

[1607] Handling emotion-based display adjustments

[1608] Step 1:

[1609] The server receives the recognized emotion data and selects the next image to display.

[1610] Step 2:

[1611] The server sends the selected image to the terminal.

[1612] Step 3:

[1613] The terminal displays the selected image to the user.

[1614] Handling Notification Functions

[1615] Step 1:

[1616] If the server identifies that a new image has been uploaded to the cloud storage and belongs to a particular person, it generates a notification.

[1617] Step 2:

[1618] The server sends a notification to the user terminal.

[1619] Step 3:

[1620] The device receives the notification and displays the message "New photo added" to the user.

[1621] Photo display and filtering process

[1622] Step 1:

[1623] The user accesses the photo viewing screen and enters search criteria (event name, date, etc.).

[1624] Step 2:

[1625] The terminal sends the search criteria to the server.

[1626] Step 3:

[1627] The server searches for relevant photos from the cloud storage based on the search criteria.

[1628] Step 4:

[1629] The server sends the search results to the user terminal.

[1630] Step 5:

[1631] The terminal receives the search results and displays them to the user.

[1632] Processing learning as we grow

[1633] Step 1:

[1634] The server periodically passes the accumulated image data to the AI ​​engine to prepare for retraining.

[1635] Step 2:

[1636] The server sends the AI ​​engine a new image dataset and instructs it to retrain.

[1637] Step 3:

[1638] The AI ​​engine retrains the facial recognition model using the new dataset.

[1639] Step 4:

[1640] The AI ​​engine returns the retrained model to the server.

[1641] Step 5:

[1642] The server uses the improved model for future facial recognition processing.

[1643] This processing step allows users to efficiently and accurately search for images of specific people and track their developmental progress. The addition of an emotion engine also enables personalized image display that reflects the user's emotions.

[1644] Example 2

[1645] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1646] In modern life, there is a need to efficiently manage large amounts of professionally taken images and easily search for images related to specific people or events. However, conventional systems require users to manually review and categorize images individually, which is time-consuming. Furthermore, they do not adjust image display based on the user's emotions, which hinders the quality of the user experience.

[1647] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for uploading images taken by a professional to cloud storage, means for automatically identifying a person's face from the uploaded image and extracting the facial features, and means for identifying a specific person based on the extracted features. This allows a user to automatically identify images of a specific person and display them in a list. The server also includes means for analyzing the user's emotions regarding the displayed image, means for selecting the next image to display based on the analyzed emotional data, and means for sending a notification when a new image is uploaded. This allows the image display to be adjusted according to the user's emotions, providing a more personalized user experience. Furthermore, the server includes means for continuously analyzing accumulated image data to learn the growth process of a specific person and means for searching and filtering images based on specific events and dates and times, allowing users to quickly and easily search for the images they are looking for.

[1648] "Professional" refers to an individual or organization that has specialized knowledge and skills in a particular field and performs work in that field as a profession.

[1649] "Image" means any representation of visual information in digital or analog form, including photographs, illustrations, graphics, etc.

[1650] "Cloud storage" refers to an online storage service for storing, managing, and backing up data over the Internet.

[1651] "Automatic face identification" refers to the process of detecting a human face in an image and identifying that face using a specific algorithm.

[1652] "Extracting facial features" refers to using a face detection algorithm to obtain the characteristic points and patterns of a person's face (e.g., eye position, nose shape, mouth size, etc.) as digital data.

[1653] "Identifying a specific person" refers to the process of identifying a person based on extracted facial features and comparing them with existing data in a database.

[1654] "Analyzing emotions" refers to the process by which the emotion engine identifies a user's emotional state (e.g., joy, surprise, sadness, etc.) based on data such as the user's facial expressions, tone of voice, and text input.

[1655] "Selecting the next image to display based on emotional data" refers to the process of using analyzed emotional data to automatically select and display an image that suits the user.

[1656] "Send notification" means that the system will notify the user through various methods (e.g., push notification, email, text message, etc.) when new images are uploaded to the cloud storage.

[1657] "Continuously analyzing stored image data" refers to periodically evaluating and analyzing stored image data and continuing to analyze it to identify new characteristics or changes in a particular person.

[1658] "Searching and filtering images based on specific events and dates and times" refers to the process of searching the system for relevant images based on specific criteria specified by the user (e.g., event name, date and time, etc.) and displaying only those images that match those criteria.

[1659] This invention relates to a system for efficiently managing a large number of images taken by professionals and adjusting the display according to the user's emotions. The system is composed of the following elements:

[1660] User terminal

[1661] server

[1662] Cloud Storage

[1663] AI Engine

[1664] Emotion Engine

[1665] First, the images taken by the professionals are uploaded to cloud storage. When the user creates an account using their device, the server validates the input information and stores it in the database, allowing the user to access the system.

[1666] The server then sends the images stored in cloud storage to an AI engine, which uses a face detection algorithm to identify people in the images and extract facial features. The server then compares the extracted features with an existing database to identify specific people. The identified images are added to a list and displayed on the user's device.

[1667] As a user browses images, the emotion engine analyzes the user's reactions (facial expressions, tone of voice, and text input) to identify their emotional state. The analyzed emotion data is sent to the server and used to select the next image to be displayed. For example, if a user expresses positive emotion toward a particular image, the server selects an image with a similar theme and instructs the user's device to display it. Conversely, if a negative emotion is recognized, the server selects an image with a different theme or person.

[1668] Additionally, when new images are uploaded to the cloud storage, the server sends a notification to the user's device, which includes a message that a new image has been added, allowing the user to immediately check the new image.

[1669] Other features of the system include a means to continuously analyze the stored image data to learn the developmental progression of a particular person, and the ability to search and filter images based on specific events or dates and times, allowing users to quickly and easily find the images they are looking for.

[1670] A specific example is shown below.

[1671] 1. A user accesses the application, creates an account, and submits their input information (e.g., "Yamada Taro, first grade elementary school student, 2023").

[1672] 2. The server validates the information, stores it in the database, and sends a notification to the user device that the account was successfully created.

[1673] 3. A professional photographer will upload photos from the April 2023 entrance ceremony to cloud storage.

[1674] 4. The server stores these photos in cloud storage and assigns metadata to each photo.

[1675] 5. The server retrieves the photo from the cloud storage and sends it to the AI ​​engine.

[1676] 6. The AI ​​engine identifies the face, extracts features, and compares them with a database to identify a specific person (e.g., "Taro Yamada").

[1677] 7. The emotion engine analyzes the user's reaction and sends the emotion data to the server.

[1678] 8. The server selects the next image to display based on the emotion data and displays it on the user's device.

[1679] 9. When a new image is uploaded, the server sends a notification and the user device displays it.

[1680] 10. The user sets search filters by event or date and time, and the server provides the corresponding images.

[1681] The following is an example of a prompt:

[1682] "Search for and display photos of Taro Yamada's 2023 entrance ceremony."

[1683] The system optimizes the user experience, allowing parents to easily browse photos from important events, and uses an emotion engine to display personalized images based on the user's emotions.

[1684] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1685] Step 1:

[1686] A user accesses an application and creates an account. The specific actions of this step are that the user enters their name, email address, password, and other required information, and clicks the submit button. The entered data (name, email address, password) is sent to the server. The server receives it and performs validation. If validation is successful, the server saves the data in the database and sends a notification to the user terminal that the account was created successfully. The user terminal displays the message "Account created." The input is the user's information, and the output is a notification that the account was created successfully.

[1687] Step 2:

[1688] The professional photographer uploads the photos he has taken to cloud storage. The specific operations of this step are that the professional photographer selects the image files taken for each event and uploads them to a cloud storage service (e.g., Amazon S3). The uploaded images are assigned metadata such as the event name, date, and location. This information is sent to a server. The server receives this information and stores it in cloud storage. The input is the image files and metadata, and the output is the images stored in cloud storage.

[1689] Step 3:

[1690] The server sends the image stored in cloud storage to the AI ​​engine. The specific operation of this step is that the server retrieves the image file from cloud storage and sends it to the AI ​​engine (e.g., Google Cloud Vision API). The input is the image file retrieved from cloud storage, and the output is the face recognition result by the AI ​​engine. The AI ​​engine uses a face detection algorithm to automatically identify the face in the image and extract facial features (e.g., eye position, nose shape, mouth size, etc.).

[1691] Step 4:

[1692] The server receives the feature data sent from the AI ​​engine and compares it with an existing database. The specific operation of this step is that the server compares the feature data with an existing person database and identifies a specific person based on the degree of match. The input is the feature data obtained from the AI ​​engine, and the output is the result of identifying a specific person. The image of the identified person is added to a list and displayed on the user's device.

[1693] Step 5:

[1694] The user device displays the image and sends the user's reaction to the emotion engine. Specific operations of this step include the user device displaying the identified image and sending the user's facial expression, tone of voice, and text input to the emotion engine (e.g., Microsoft Azure Emotion API). The input is the user's reaction data, and the output is the emotion data identified by the emotion engine.

[1695] Step 6:

[1696] The emotion engine analyzes the user's emotions and sends the results to the server. As a specific operation of this step, the emotion engine analyzes the user's reaction data and identifies the emotional state (e.g., joy, surprise, sadness, etc.). The input is the user's reaction data, and the output is the analyzed emotion data. The server receives this data and selects the next image to display.

[1697] Step 7:

[1698] The server selects the next image to display based on the emotion data and instructs the user terminal. As a specific operation of this step, the server generates a list of images to display next based on the analyzed emotion data. For example, if the user expresses positive emotion toward a particular image, the server selects images with a similar theme. The input is the analyzed emotion data, and the output is a list of images to display next. The user terminal displays the images according to this instruction.

[1699] Step 8:

[1700] The server sends a notification when a new image is uploaded. The specific operation of this step is that the server monitors the cloud storage and sends a notification to the user device when a new image is uploaded. The input is the newly uploaded image, and the output is the notification to the user device. The user device displays a notification saying "A new photo has been added."

[1701] Step 9:

[1702] The user sets a search filter based on an event or date and time, and the server provides the relevant images. In this step, the user enters search criteria based on a specific event or date and time within the application and presses the search button. The input is the search filter set by the user, and the output is a list of relevant images. The server searches for relevant images based on this request and sends the results to the user's device. The user's device displays the search results.

[1703] Example prompt sentence:

[1704] "Search and view photos from the 2023 Commencement Ceremony."

[1705] (Application example 2)

[1706] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1707] Existing image management systems are required to efficiently classify images taken by professionals and provide a means to easily search and display the images they need. However, the lack of a function to dynamically change the display content based on user sentiment limits the ability to improve the user experience. Furthermore, it is often difficult for end users to effectively search and filter images based on specific events or dates and times. A new system that solves this problem is needed.

[1708] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading images taken by a professional to cloud storage, means for automatically identifying people's faces from the uploaded images and extracting their facial features, means for identifying specific people based on the extracted features, means for displaying a list of images of the identified specific people, means for sending a notification when a new image is uploaded, means for recognizing a user's facial expression in real time and identifying their emotion, means for adjusting the next image to be displayed based on the identified emotion, means for continuously analyzing accumulated image data to learn the growth process of a specific person, and means for searching and filtering images based on specific events and dates and times. This allows the server to provide display content optimized based on the user's emotions, making it easy for end users to search and filter for the images they need.

[1709] "Professionally captured images" are high-quality images captured using specialized techniques and equipment.

[1710] "Cloud storage" refers to external data storage services accessible over the Internet that allow users to store and manage data remotely.

[1711] "Means for automatically identifying a person's face" is a technology that detects a person's face in an image and automatically recognizes the position and outline of that face.

[1712] "Means for extracting facial features" refers to a technology that analyzes features such as the eyes, nose, and mouth from a person's face and extracts them as data.

[1713] The "means for identifying a specific person" is a technology that recognizes a specific person by comparing the extracted facial features with facial data registered in a database.

[1714] The "means for displaying images in a list" is a technology that visually organizes images related to an identified person and displays them all together on the screen.

[1715] "Means for sending notifications" refers to technology that sends alerts or messages to user terminals when new information is added.

[1716] "Means for recognizing a user's facial expression in real time" refers to a technology that captures the user's face with a camera, instantly analyzes their facial expression, and determines their emotional state.

[1717] The "means for identifying emotions" is a technology that determines the user's emotions (for example, joy, surprise, sadness, etc.) based on analyzed facial expression data.

[1718] The "means for adjusting the next image to be displayed" is a technique for changing the order or content of images to be presented to the user based on the recognized emotion.

[1719] "Means for learning growth processes" refers to technology that analyzes image data accumulated over time to learn about the growth and changes of a specific person.

[1720] "Means for searching and filtering images" refers to a technique for specifying images based on specific conditions (for example, event name or date and time) and extracting images that meet the purpose.

[1721] The system of the present invention efficiently manages images taken by professionals and improves the user experience by dynamically displaying images according to the user's emotions. This system mainly consists of the following elements.

[1722] Server, user terminal, cloud storage, AI engine, and emotion engine.

[1723] 1. User Registration

[1724] When a user first accesses the application, the user creates an account on the user's device. During this process, the user enters basic information. The server receives this information, validates it, and then saves it in the database to create the account.

[1725] 2. Upload a photo

[1726] Professionally taken images are uploaded to cloud storage. The server receives these images and stores them in cloud storage along with metadata (e.g., event name, date, location).

[1727] 3. AI-powered facial recognition and classification

[1728] The server passes the images stored in cloud storage to an AI engine, which uses a face detection algorithm to identify faces in the photos, identifies specific people based on the extracted facial features, and stores the results in a database.

[1729] 4. Emotion Recognition by Emotion Engine

[1730] The user device analyzes the user's reaction to the displayed images in real time through an emotion engine, which analyzes the user's facial expressions and tone of voice to identify emotions.

[1731] 5. Adjusting display based on emotions

[1732] The server then adjusts the next image to be displayed based on the recognized emotion: if a positive emotion is detected, images with related themes are prioritized; if a negative emotion is detected, images with different themes or people are displayed instead.

[1733] 6. Notification function

[1734] When a new image is uploaded to cloud storage and is recognized by the AI ​​engine as an image of a specific person, the server sends a notification to the user's device, allowing the user to immediately view the new image.

[1735] 7. Viewing and filtering photos

[1736] Users can access the photo browsing screen within the application and filter photos based on events or dates. The server generates a list of photos based on the user's request and sends it to the user's device, allowing the user to visually view the photos.

[1737] 8. Learning the process of growth

[1738] The server continuously trains the AI ​​engine based on the stored image data, tracking the development of specific individuals and improving the accuracy of identification with each new data point.

[1739] Hardware and software used

[1740] The following hardware and software are specifically used to implement this system:

[1741] Hardware:

[1742] 1. Smart glasses (e.g., Google Glass)

[1743] 2. Camera

[1744] software:

[1745] 1. OpenCV (for face recognition)

[1746] 2. EmotionRecognizer (emotion recognition library)

[1747] 3. ProductRecommender (product recommendation engine)

[1748] Specific examples

[1749] As a user walks through a virtual store wearing smart glasses, the camera in the smart glasses captures the user's face and analyzes their facial expressions in real time. For example, if a user smiles after looking at a product, the emotion engine will identify that positive emotion and instantly recommend and display relevant products. In this way, it is possible to suggest products tailored to the user's emotional state.

[1750] Prompt example (text format)

[1751] Design a program that allows smart glasses to recognize the user's facial expressions in real time as they walk through a virtual store and display products according to their emotions. Include the following elements:

[1752] 1. Detect faces from the camera stream.

[1753] 2. Uses facial recognition and emotion identification algorithms.

[1754] 3. If positive sentiment is detected, product recommendations are displayed.

[1755] 4. Specify the hardware and software you will be using.

[1756] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1757] Step 1:

[1758] When a user walks through the virtual store, the user device (smart glasses) takes a picture of the user's face in real time using the built-in camera. As input, it takes the video frames captured by the camera and passes them to the next processing step. As output, it obtains the captured video frames.

[1759] Step 2:

[1760] The server receives the video frame and runs a face detection algorithm using OpenCV. The pixel data of the video frame is passed as input, and the area containing the face is identified. The data is then processed to extract the face position and bounding box. The output is a video frame containing face position information.

[1761] Step 3:

[1762] Based on the identified facial area, the server uses EmotionRecognizer to analyze facial expressions and identify emotions. The input is the position information of the face and the pixel data of that area. The data is then processed by analyzing facial features (e.g., eye shape and mouth open / closed degree) and classifying the emotion (happiness, surprise, etc.). The output is the identified emotion data.

[1763] Step 4:

[1764] The server receives the emotion data and adjusts the next image to be displayed based on the identified emotion. The emotion data and metadata of the products in the virtual store are passed as input. The data is then processed to select products that match the emotion data and generate a list of highly relevant products. The output is the selected product list.

[1765] Step 5:

[1766] Based on the product list, the server sends the product information to be displayed next to the user's terminal. The selected product list is passed as input. Data processing involves packaging the product information in a format appropriate for the user's terminal. The packaged product information is obtained as output.

[1767] Step 6:

[1768] The user terminal displays the product on the smart glasses display based on the received product information. The product information sent from the server is passed as input. Specific operations include displaying product images and explanatory text on the display, visually presenting the product to the user. The output is a product display that the user can visually confirm.

[1769] Through the above processing steps, product suggestions based on the user's emotions are realized in real time, and a system that enhances the user experience is constructed.

[1770] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1771] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1772] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1773] [Fourth embodiment]

[1774] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1775] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1776] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1777] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1778] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1779] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1780] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1781] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1782] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1783] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1784] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1785] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1786] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1787] ---

[1788] This invention is a system for automatically classifying images and tracking the growth process of specific individuals. This system uploads professionally taken images to cloud storage, uses an AI engine to identify faces, extracts their features, and displays a list of images of the identified specific individuals.

[1789] System Configuration

[1790] The system mainly consists of the following components:

[1791] User terminal

[1792] server

[1793] Cloud Storage

[1794] AI Engine

[1795] User Registration

[1796] When a user first accesses the application, the user creates an account on the user's device. During this process, the user enters basic information such as an email address, password, child's name, and school year. The server receives this information, validates it, and stores it in the database. This creates a new account and sends the user a notification that the account was created successfully.

[1797] Upload a photo

[1798] Professionally taken images are uploaded to cloud storage. The professional photographer compiles a large number of photos taken at events (e.g., athletic meets, entrance ceremonies, etc.) and adds metadata (event name, date, location, etc.) to each photo. The server receives these images and stores them in cloud storage along with the metadata.

[1799] AI-powered facial recognition and classification

[1800] The server passes the images stored in cloud storage to the AI ​​engine, which uses a face detection algorithm to identify faces in the photos. During this process, facial features (such as eye position, nose shape, and mouth size) are extracted. The server then compares these features with facial data in an existing database and identifies specific people based on the degree of match. The identified images are added to a list of images that are determined to contain specific people.

[1801] Notification function

[1802] When a new image is uploaded to the cloud storage and is recognized by the AI ​​as an image of a specific person, the server sends a notification to the user's device. This notification includes information that a new image has been added. The user's device receives this notification and notifies the user via a pop-up message or other means.

[1803] Viewing and filtering photos

[1804] Users can access the photo browsing screen within the application and set filtering criteria to search for photos based on specific events or dates. The server generates a list of photos based on the user's request and sends it to the user's device, which then visually displays the received photos.

[1805] Learning the process of growth

[1806] The server continuously trains the AI ​​engine based on the accumulated image data. This process tracks the development of specific individuals and improves the accuracy of identification with each new data set. The new data set is used to retrain the facial recognition model, optimizing the overall performance of the system.

[1807] Specific examples

[1808] User Registration

[1809] 1. The user accesses the application and enters the required information on the account creation screen (e.g., Yamada Taro, first grade elementary school student in 2023).

[1810] 2. The server retrieves this information, validates it, and if successful, saves it in the database and creates an account.

[1811] 3. The user device will display a message indicating that the account was created successfully.

[1812] Upload a photo

[1813] 1. A professional photographer will compile photos from the April 2023 entrance ceremony and upload them to cloud storage.

[1814] 2. The server stores these photos in cloud storage and assigns metadata to each photo.

[1815] AI-powered facial recognition and classification

[1816] 1. The server passes the uploaded photo to the AI ​​engine and performs facial recognition.

[1817] 2. The AI ​​engine identifies faces in each photo and extracts features.

[1818] 3. The server identifies a specific person from the database based on the features and adds the image to a list based on the degree of match (e.g., identifying it as a photo of Taro Yamada).

[1819] Notification function

[1820] 1. When the server identifies a new photo as belonging to a specific person, it sends a notification to the user's device.

[1821] 2. The user device displays the message "New photo added."

[1822] Viewing and filtering photos

[1823] 1. The user accesses the photo viewing screen, sets a filter such as "Events: Entrance Ceremony 2023," and performs a search.

[1824] 2. The server searches for the relevant photos and sends the results to the user's device.

[1825] 3. The user terminal displays the search results to the user.

[1826] This allows users to efficiently search for photos based on specific events or dates. Furthermore, as the AI ​​engine continues to learn, its identification accuracy improves, making it easier to track the growth process of a specific person. This system prevents parents from overlooking photos of important events and makes it easier to search for commemorative photos.

[1827] The processing flow will be explained below.

[1828] Handling user registration

[1829] Step 1:

[1830] The user accesses the application and enters the required information (email address, password, child's name, grade, etc.) on the account creation screen.

[1831] Step 2:

[1832] The terminal transmits the input information to the server.

[1833] Step 3:

[1834] The server validates the information received and checks the format and content of the input information.

[1835] Step 4:

[1836] The server saves the validated information to the database and creates the new account.

[1837] Step 5:

[1838] The server notifies the device that the account creation was successful.

[1839] Step 6:

[1840] The device displays a message to the user confirming successful account creation.

[1841] Handling photo uploads

[1842] Step 1:

[1843] A professional photographer takes images from the event and prepares them for upload to cloud storage, where they are tagged with metadata (event name, date, location, etc.).

[1844] Step 2:

[1845] The device sends the image with the metadata to the server.

[1846] Step 3:

[1847] The server stores the received image data in cloud storage.

[1848] AI-powered facial recognition and classification

[1849] Step 1:

[1850] The server prepares the images stored in cloud storage to be passed to the AI ​​engine.

[1851] Step 2:

[1852] The server sends the image data to the AI ​​engine and instructs it to begin facial recognition processing.

[1853] Step 3:

[1854] The AI ​​engine uses a face detection algorithm to identify faces in the image.

[1855] Step 4:

[1856] The AI ​​engine extracts facial features (eye position, nose shape, mouth size, etc.).

[1857] Step 5:

[1858] The server compares the extracted features with face data in an existing database and calculates the degree of match.

[1859] Step 6:

[1860] The server identifies the specific person based on the degree of match and adds them to a photo list of the specific person.

[1861] Handling Notification Functions

[1862] Step 1:

[1863] The server generates a notification when a new image is identified as being of a particular person.

[1864] Step 2:

[1865] The server sends a notification to the user terminal.

[1866] Step 3:

[1867] The device receives the notification and displays the message "New photo added" to the user.

[1868] Photo display and filtering process

[1869] Step 1:

[1870] The user accesses the photo viewing screen and enters search criteria (e.g., event name, date, etc.).

[1871] Step 2:

[1872] The terminal transmits the entered search conditions to the server.

[1873] Step 3:

[1874] Based on the search conditions received by the server, the server searches for corresponding photos from the cloud storage.

[1875] Step 4:

[1876] The server sends the search results to the user terminal.

[1877] Step 5:

[1878] The device receives the search results, organizes them visually, and displays them to the user.

[1879] Processing learning as we grow

[1880] Step 1:

[1881] The server periodically passes the accumulated image data to the AI ​​engine to prepare for retraining.

[1882] Step 2:

[1883] The server sends the AI ​​engine a new image dataset and instructs it to retrain.

[1884] Step 3:

[1885] The AI ​​engine retrains the facial recognition model using the new dataset.

[1886] Step 4:

[1887] The AI ​​engine returns the retrained model to the server.

[1888] Step 5:

[1889] The server uses the improved model for future facial recognition processing.

[1890] These are the specific processing steps of this system, allowing users to quickly and accurately find images of specific people and efficiently track their growth process.

[1891] Example 1

[1892] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1893] Modern digital photo management requires efficient classification of large amounts of photo data and rapid retrieval of photos related to specific people or events. However, manual classification and retrieval is laborious and time-consuming. Improving identification accuracy is particularly important when tracking the growth of specific people. Conventional technologies have not adequately addressed these challenges, and further improvements in the user experience are needed.

[1894] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1895] In this invention, the server includes a means for an end user to input information and create an account, a means for uploading images taken by a professional to network storage, a means for automatically identifying people's faces from the uploaded images and extracting facial features, a means for sending a notification to a user terminal when a new image of a specific person is uploaded, and a means for displaying or filtering the identified images of the specific person based on search conditions. This makes it possible to quickly and accurately search for images related to a specific person or event from a large amount of photo data, and also to track their growth.

[1896] An "end user" is the final user of the system, who performs operations such as uploading, searching, and viewing photos.

[1897] An "expert" is someone who has the skills and knowledge to take photographs and can provide high-quality images.

[1898] "Network storage" is an online storage device accessible via the Internet, and is a system for saving and managing data.

[1899] "Automatic face identification" is the process of using image analysis technology to identify the faces of people present in a photograph.

[1900] "Features" refer to the unique attributes and shape information required to identify a face, and by extracting these, individual people can be recognized.

[1901] A "specific person" is an individual identified based on facial features registered in a database.

[1902] A "user terminal" is a device that connects to the system and operates it, and includes smartphones, tablets, PCs, etc.

[1903] A "notification" is a message from the system to inform the user of new information or events.

[1904] "Search conditions" are filters and keywords that a user sets when searching for a specific photo, and include the event name, date and time, etc.

[1905] "Growth tracking" is the process of analyzing accumulated image data to track the changes in a particular person over the years.

[1906] The present invention provides a system for automatically classifying images and tracking the growth process of a specific person. The system includes a means for an end user to input information and create an account, a means for uploading images taken by a professional to a network storage, a means for automatically identifying faces of people from the uploaded images and extracting facial features, a means for sending a notification to a user terminal when a new image of the specific person is uploaded, and a means for displaying or filtering the identified images of the specific person based on search criteria.

[1907] Hardware and software used

[1908] The system mainly consists of the following components:

[1909] User devices: smartphones, tablets, computers, etc.

[1910] Server: High-performance computing server

[1911] Network storage: Cloud storage services (e.g., Amazon S3 or Google Cloud Storage)

[1912] AI Engine: Machine learning models (e.g., TensorFlow or OpenCV) for face recognition and feature extraction

[1913] Example of user registration

[1914] A user accesses an application and creates an account. For example, the user enters the email address "user@example.com", the password "securepassword", the child's name "Child A", and the grade "First grade of elementary school". The device sends this information to the server, which validates the information. The server then saves the information in the database and a new account is created. The user's device displays a message indicating that the account was created successfully.

[1915] Example of photo upload

[1916] An expert (e.g., a professional photographer) takes photos of an event (e.g., an entrance ceremony) and uploads them to network storage. When uploading, the photos are given metadata such as the event name "Entrance Ceremony 2023," the date "April 1, 2023," and the location "Elementary School A." The server receives these photos and metadata and stores them in network storage.

[1917] Specific examples of AI facial recognition and classification

[1918] The server periodically checks the network storage and detects new photos. It passes the new photos to the AI ​​engine for facial recognition processing. The AI ​​engine uses a face detection algorithm to identify faces in the photos and extract features such as eye position, nose shape, and mouth size. The server compares these features with facial data in an existing database and identifies a specific person (e.g., Child A) based on the degree of match. The server adds the identification results to a list of photos of specific people.

[1919] Examples of notification features

[1920] When the server identifies a new photo of a specific person (e.g., Child A), it sends a notification to the user device saying, "A new photo has been added." The user device receives this notification and notifies the user via a pop-up message or similar.

[1921] Examples of photo display and filtering

[1922] The user accesses the photo viewing screen within the application and sets filtering conditions for a specific event, such as "Entrance Ceremony 2023," or a date, such as "April 1, 2023." The server searches for matching photos based on the user's request and generates results. The user's device displays a list of search results.

[1923] Examples of learning in the process of growth

[1924] The server continuously trains the AI ​​engine based on the accumulated image data. For example, the AI ​​engine retrains based on past photos of Child A, and improves its recognition accuracy by uploading new photos.

[1925] Example prompts for generative AI models

[1926] "Please explain the user registration process in detail, from when the user enters the required information, through when the server validates and saves the information to the database, and notifies the user that the account was successfully created."

[1927] "Please explain in detail the process of uploading photos. Please explain in detail the process from an expert uploading photos to network storage to the server storing them with metadata."

[1928] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1929] Step 1: User Registration

[1930] 1. The user accesses the application and enters the required information (email address, password, child's name, grade, etc.) on the account creation screen.

[1931] Input: Email address "user@example.com", password "securepassword", child's name "Child A", grade "First grade of elementary school"

[1932] Specific operation: The user enters information using the input form and presses the "Register" button.

[1933] 2. The device sends the entered information to the server and requests account creation.

[1934] Output: Data packet containing user information

[1935] Specific operation: The terminal generates a data packet and sends it to the server.

[1936] 3. Validate the information received by the server (check the format of the email address, verify the strength of the password, etc.).

[1937] Input: Data packet containing user information

[1938] Output: Validation result (pass or fail)

[1939] What happens: The server checks the format of the email address and evaluates the strength of the password. If validation is successful, it proceeds to the next step.

[1940] 4. If the server validates successfully, it saves the information in the database and creates a new account.

[1941] Input: User information that has been successfully validated

[1942] Output: Account information stored in the database

[1943] Specific operation: The server opens a database connection and registers the user information as a new record.

[1944] 5. Send a notification to the device that the account was created successfully.

[1945] Input: Account creation success status

[1946] Output: Account creation successful message

[1947] Specific operation: The server generates a success message and sends it to the terminal, which displays the message.

[1948] Step 2: Upload a photo

[1949] 1. Professionals take photos at the event and upload them to network storage.

[1950] Input: Photo file

[1951] Output: Upload request to network storage

[1952] What happens: The expert uses the upload interface to select a photo file and begin uploading.

[1953] 2. An expert assigns metadata to each photo (e.g., event name "Entrance Ceremony 2023", date "April 1, 2023", location "Elementary School A").

[1954] Input: Photo file, metadata

[1955] Output: Photo files with metadata

[1956] Specific operation: The expert enters the required information into the metadata input form and attaches the metadata to the photo.

[1957] 3. The server receives the uploaded photos and metadata and stores them in network storage.

[1958] Input: Photo files with metadata

[1959] Output: Photos stored on network storage

[1960] Specific operation: The server connects to network storage and stores photos and metadata.

[1961] Step 3: AI-powered facial recognition and classification

[1962] 1. The server periodically checks the cloud storage for new photos.

[1963] Input: Photo data from cloud storage

[1964] Output: A list of new photos

[1965] Specific operation: The server sets a timer and periodically accesses the cloud storage to check for new photo files.

[1966] 2. The server passes the new photo to the AI ​​engine for facial recognition processing.

[1967] Input: New photo file

[1968] Output: Face recognition results

[1969] Specific operation: The server passes the photo file to the AI ​​engine, which runs the face detection algorithm.

[1970] 3. The AI ​​engine uses a face detection algorithm to identify faces in the photo and extract their features.

[1971] Input: Photo file

[1972] Output: Facial feature data

[1973] Specific operation: The AI ​​engine detects the face and extracts features such as eye position, nose shape, and mouth size.

[1974] 4. The server compares the features with facial data in an existing database and identifies the person based on the degree of match.

[1975] Input: Facial feature data

[1976] Output: Identification result of a specific person

[1977] Specific operation: The server compares the feature data with face data in the database, calculates the degree of match, and identifies the person.

[1978] 5. The server adds the new photo to the photo list of the identified person.

[1979] Input: Identification result of a specific person

[1980] Output: Updated list of photos of specific people

[1981] Specific operation: The server adds the photo to the list of specific people and updates the list.

[1982] Step 4: Notifications

[1983] 1. The server sends a notification to the user device when a new photo of a specific person is identified.

[1984] Input: Identification results for new photos of a specific person

[1985] Output: Notification message

[1986] Specific operation: The server generates a notification message and sends it to the user terminal.

[1987] 2. The user device receives the notification and notifies the user via a pop-up message or similar.

[1988] Input: Notification message

[1989] Output: Display a popup message

[1990] Specific operation: The user terminal receives the notification message and displays a pop-up message.

[1991] Step 5: View and filter photos

[1992] 1. The user accesses the photo viewing screen within the application and sets filtering criteria for specific events or dates.

[1993] Input: Filtering criteria such as event name and date

[1994] Output: Search request based on filtering criteria

[1995] Specific operation: The user enters conditions into the filtering form and presses the search button.

[1996] 2. The server searches for relevant photos based on the user's request and generates the results.

[1997] Input: Filtering criteria

[1998] Output: A list of photos as search results

[1999] Specific operation: The server searches the photos in the database and generates a list of photos that match the criteria.

[2000] 3. The user device receives the search results and displays a list of photos.

[2001] Input: Photo list as search results

[2002] Output: List of photos

[2003] Specific operation: The user terminal loads the photo list onto the display screen and displays a visual list.

[2004] Step 6: Learning the process of growth

[2005] 1. The server continuously trains the AI ​​engine based on the accumulated image data.

[2006] Input: Stored image data

[2007] Output: Updated AI model

[2008] Specific operation: The server supplies image data to the AI ​​engine and causes it to re-learn.

[2009] 2. The server uses the new dataset to retrain the facial recognition model, improving the system's overall recognition accuracy.

[2010] Input: New dataset, existing AI model

[2011] Output: Retrained AI model

[2012] Specific operation: The server retrains the AI ​​model using a new dataset to improve its recognition accuracy.

[2013] (Application example 1)

[2014] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2015] Providing personalized services to modern consumers is extremely important in increasing customer satisfaction. However, it is difficult to quickly and accurately grasp customer attributes in physical stores and provide services that are appropriate for the customer on the spot. As a result, customer needs cannot be adequately met and store profits are not maximized. It is also difficult to provide a consistent customer experience to multiple customers who visit a store. To solve these issues, a means is needed to use advanced technology to provide personalized services based on customer attributes.

[2016] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2017] In this invention, the server includes means for uploading images taken by a professional to cloud storage, means for automatically identifying people's faces from the uploaded images and extracting their facial features, means for identifying specific people based on the extracted features, means for displaying a list of images of the identified specific people, means for sending a notification when a new image is uploaded, means for identifying the faces of customers visiting the store and specifying the customer's attributes, and means for providing personalized services based on the specified customer attributes. This makes it possible to easily determine customer attributes even in physical stores and provide optimal services to each customer.

[2018] A "professional" is someone who has specialized skills and knowledge and makes a living from them.

[2019] "Cloud storage" refers to remote servers that provide services for storing and managing data over the Internet.

[2020] "Means for identifying human faces" refers to methods and devices that use cameras and image analysis software to identify and recognize human faces in images.

[2021] "Means for extracting facial features" refers to a method of analyzing and acquiring features such as facial shape, pattern, and location information using a facial recognition algorithm.

[2022] The "means for identifying a specific person" refers to a method and apparatus that uses the extracted facial features to match existing facial data in a database and identify matching people.

[2023] "Means for displaying in a list" refers to a method for displaying images of identified specific persons together on a display or screen in a visually easy-to-understand format.

[2024] "Means of sending notifications" refers to the methods of sending messages or alerts to users when new images are uploaded.

[2025] "Means for identifying the faces of customers visiting a store" refers to a method and device for photographing the faces of customers using a camera or other device installed in a physical store and recognizing the faces of customers from the images.

[2026] "Means for identifying customer attributes" refers to a method for analyzing and identifying a customer's age, gender, and other attribute information based on recognized facial features.

[2027] The "means for providing personalized services" refers to a method and apparatus for providing products and services optimized for a customer based on the identified customer's attribute information.

[2028] This invention is a system that identifies customers' faces in brick-and-mortar stores and provides personalized services based on their attributes. The system is mainly composed of a server, cloud storage, an AI engine, and a user's device. The specific configuration and operation of this system are described below.

[2029] User Registration

[2030] When a user first accesses the system through a smartphone application, they create an account. They enter their email address, password, and other basic information, and the server receives, validates, and stores this information in a database.

[2031] Upload a photo

[2032] It provides a way for professionally taken images to be uploaded to cloud storage, where the professional adds metadata such as the event name, date, and location to each photo, and the server receives and stores the metadata in the cloud storage.

[2033] AI-powered facial recognition and classification

[2034] The server passes the images stored in cloud storage to an AI engine for facial identification, which uses a facial recognition algorithm to extract facial features and match them with an existing database to identify a specific person.

[2035] Notification function

[2036] When a new image is uploaded to the cloud storage and identified by the AI ​​engine as belonging to a specific person, the server sends a notification to the user's device, which includes information that a new image has been added.

[2037] Viewing and filtering photos

[2038] Users can access the photo browsing screen in the application and search for photos based on specific events or dates and times. The server searches for the relevant photos based on the user's request and sends them to the user's device.

[2039] Identifying the faces of customers visiting a store

[2040] Using cameras and smartphones installed in physical stores, the server photographs and recognizes the faces of customers visiting the store, extracts their facial features, and uses that information to identify their attributes (age, gender, etc.).

[2041] Providing personalized service

[2042] The server customizes the store's services based on the identified customer attributes, for example, by providing recommended products and specific campaign information.

[2043] Specific examples

[2044] 1. User Registration:

[2045] A user accesses the application and enters basic information.

[2046] The server receives the information and stores it in a database.

[2047] 2. Upload a photo:

[2048] Professionally captured images are uploaded to cloud storage.

[2049] The server receives the images and stores them along with the metadata.

[2050] 3. Identifying the faces of customers visiting your store:

[2051] Cameras installed at the entrances of physical stores capture customers' faces.

[2052] The server analyzes facial features and identifies customer attributes.

[2053] 4. Service Provision:

[2054] The server provides the optimal service to the customer based on the identified customer attribute information.

[2055] Prompt Sentence Examples

[2056] "How can we provide specific services that are best suited to store customers based on facial recognition results? Please provide the following information:

[2057] Facial image

[2058] Customer attributes (age, gender)

[2059] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2060] Step 1:

[2061] A user creates an account through a smartphone application. The user enters their email address, password, and other basic information. The information is sent to the server, where it is validated and stored in a database.

[2062] Input: User's email address, password, basic information

[2063] Data processing: information validation

[2064] Output: Account information is saved in the database

[2065] Step 2:

[2066] Professionally taken images are uploaded to cloud storage, and metadata such as the event name, date, and location are added to the images. The server receives these and stores them in cloud storage.

[2067] Input: Professionally captured images and metadata

[2068] Data Computing: Saving images and metadata to cloud storage

[2069] Output: Images and metadata are saved to cloud storage

[2070] Step 3:

[2071] The server passes the images stored in cloud storage to an AI engine that identifies faces, which uses a facial recognition algorithm to extract facial features and match them with existing databases.

[2072] Input: Images stored in cloud storage

[2073] Data Computing: Facial feature extraction and database matching

[2074] Output: Information about the specific person identified

[2075] Step 4:

[2076] When a new image is uploaded and the AI ​​engine identifies a specific person, the server sends a notification to the user's device, which includes information that a new image has been added.

[2077] Input: Information that identifies a specific person

[2078] Data processing: Notification message generation

[2079] Output: Notification to user terminal

[2080] Step 5:

[2081] The user accesses the photo viewing screen within the application and searches for photos based on a specific event or date and time. The server searches for the corresponding photos based on the user's request and sends them to the user's device.

[2082] Input: User request (event name and date and time)

[2083] Data calculation: Search for relevant photos

[2084] Output: Send search results to the user's terminal

[2085] Step 6:

[2086] A camera installed at the entrance of a physical store captures the faces of customers. The server receives the captured images, analyzes their facial features, and identifies the customer's attributes (age, gender).

[2087] Input: An image of the customer captured by a camera

[2088] Data Computing: Facial feature analysis and attribute identification

[2089] Output: Customer attribute information

[2090] Step 7:

[2091] The server customizes the store's services based on the identified customer attribute information, for example, by recommending specific products or providing specific campaign information.

[2092] Input: Customer attribute information

[2093] Data processing: Creating customized service content

[2094] Output: personalized service presentation

[2095] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2096] ---

[2097] The system uploads professionally taken images to cloud storage, uses an AI engine to identify faces, extracts their features to recognize specific people, and uses an additional emotion engine to identify the user's emotions, thereby personalizing the images displayed. The system consists of the following components:

[2098] User terminal

[2099] server

[2100] Cloud Storage

[2101] AI Engine

[2102] Emotion Engine

[2103] User Registration

[2104] When a user first accesses the application, the user creates an account on the user's device. During this process, the user enters basic information such as an email address, password, child's name, and school year. The server receives this information, validates it, and stores it in the database. This creates a new account and sends the user a notification that the account was created successfully.

[2105] Upload a photo

[2106] Professionally taken images are uploaded to cloud storage. The professional photographer compiles a large number of photos taken at the event and adds metadata (event name, date, location, etc.) to each photo. The server receives these images and stores them in cloud storage along with the metadata.

[2107] AI-powered facial recognition and classification

[2108] The server passes the images stored in cloud storage to the AI ​​engine, which uses a face detection algorithm to identify faces in the photos. During this process, facial features (such as eye position, nose shape, and mouth size) are extracted. The server then compares these features with facial data in an existing database and identifies specific people based on the degree of match. The identified images are added to a list of images that are determined to contain specific people.

[2109] Emotion recognition by emotion engine

[2110] The user device analyzes the user's reaction to the displayed image through an emotion engine, which analyzes the user's facial expressions, tone of voice, text input, etc. to identify emotions (happiness, surprise, sadness, etc.).

[2111] Emotion-based display adjustment

[2112] Based on the perceived emotion, the device will adjust the next image displayed: for example, if the user expresses positive emotion toward a particular image, it will continue to display images with a similar theme, or if negative emotion toward a particular image is detected, it will switch to images with a different theme or person.

[2113] Notification function

[2114] When a new image is uploaded to the cloud storage and is recognized by the AI ​​as an image of a specific person, the server sends a notification to the user's device that a new image has been added, allowing the user to immediately view the new image.

[2115] Viewing and filtering photos

[2116] Users can access the photo browsing screen within the application and set filtering criteria to search for photos based on specific events or dates. The server generates a list of photos based on the user's request and sends it to the user's device, which then visually displays the received photos.

[2117] Learning the process of growth

[2118] The server continuously trains the AI ​​engine based on the accumulated image data. This process tracks the development of specific individuals and improves the accuracy of identification with each new data set. The new data set is used to retrain the facial recognition model, optimizing the overall performance of the system.

[2119] Specific examples

[2120] User Registration

[2121] 1. The user accesses the application and enters the required information on the account creation screen (e.g., Yamada Taro, first grade elementary school student in 2023).

[2122] 2. The server retrieves this information, validates it, and if successful, saves it in the database and creates an account.

[2123] 3. The user device will display a message indicating that the account was created successfully.

[2124] Upload a photo

[2125] 1. A professional photographer will compile photos from the April 2023 entrance ceremony and upload them to cloud storage.

[2126] 2. The server stores these photos in cloud storage and assigns metadata to each photo.

[2127] AI-powered facial recognition and classification

[2128] 1. The server passes the uploaded photo to the AI ​​engine and performs facial recognition.

[2129] 2. The AI ​​engine identifies faces in each photo and extracts features.

[2130] 3. The server identifies a specific person from the database based on the features and adds the image to a list based on the degree of match (e.g., identifying it as a photo of Taro Yamada).

[2131] Emotion recognition by emotion engine

[2132] 1. The emotion engine analyzes the user's reaction to the displayed image (facial expression, tone of voice, text input).

[2133] 2. The emotion engine identifies the user's emotion and sends the data to the server.

[2134] Emotion-based display adjustment

[2135] 1. The server selects the next image to display based on the recognized emotion.

[2136] 2. If a user expresses positive feelings about a particular image, the device is instructed to display images with a similar theme.

[2137] 3. If the user expresses negative emotions, instruct the device to switch to an image of a different subject or person.

[2138] Notification function

[2139] 1. When the server identifies a new photo as belonging to a specific person, it sends a notification to the user's device.

[2140] 2. The user device displays the message "New photo added."

[2141] Viewing and filtering photos

[2142] 1. The user accesses the photo viewing screen, sets a filter such as "Events: Entrance Ceremony 2023," and performs a search.

[2143] 2. The server searches for the relevant photos and sends the results to the user's device.

[2144] 3. The user terminal displays the search results to the user.

[2145] This allows users to efficiently search for photos based on specific events or dates. Furthermore, as the AI ​​engine continues to learn, its identification accuracy improves, making it easier to track the growth process of a specific person. The addition of an emotion engine further improves the user experience by identifying the user's emotions and personalizing the images displayed. This system prevents parents from overlooking photos of important events and makes it easier to search for and display commemorative photos.

[2146] The processing flow will be explained below.

[2147] Handling user registration

[2148] Step 1:

[2149] The user accesses the application and enters the required information (email address, password, child's name, grade, etc.) on the account creation screen.

[2150] Step 2:

[2151] The terminal transmits the input information to the server.

[2152] Step 3:

[2153] The server validates the information received and checks the format and content of the input information.

[2154] Step 4:

[2155] The server saves the validated information to the database and creates the new account.

[2156] Step 5:

[2157] The server notifies the device that the account creation was successful.

[2158] Step 6:

[2159] The device displays a message to the user confirming successful account creation.

[2160] Handling photo uploads

[2161] Step 1:

[2162] A professional photographer prepares images taken at an event (e.g., athletic meet, entrance ceremony, etc.) for uploading to cloud storage.

[2163] Step 2:

[2164] A professional photographer adds metadata (event name, date, location, etc.) to the images.

[2165] Step 3:

[2166] The device sends the image with the metadata to the server.

[2167] Step 4:

[2168] The server stores the received image data in cloud storage.

[2169] AI-powered facial recognition and classification

[2170] Step 1:

[2171] The server prepares the images stored in cloud storage to be passed to the AI ​​engine.

[2172] Step 2:

[2173] The server sends the image data to the AI ​​engine and instructs it to begin facial recognition processing.

[2174] Step 3:

[2175] The AI ​​engine uses a face detection algorithm to identify faces in the image.

[2176] Step 4:

[2177] The AI ​​engine extracts facial features (eye position, nose shape, mouth size, etc.).

[2178] Step 5:

[2179] The server compares the extracted features with face data in an existing database and calculates the degree of match.

[2180] Step 6:

[2181] The server identifies the specific person based on the degree of match and adds them to a list of photos of the specific person.

[2182] Emotion recognition processing by emotion engine

[2183] Step 1:

[2184] The user terminal transmits the user's facial expression, tone of voice, text input, etc. in response to the displayed image to the emotion engine.

[2185] Step 2:

[2186] The emotion engine analyzes the user's emotions and generates emotion data (e.g., joy, surprise, sadness, etc.).

[2187] Step 3:

[2188] The emotion data generated by the emotion engine is sent to the server.

[2189] Handling emotion-based display adjustments

[2190] Step 1:

[2191] The server receives the recognized emotion data and selects the next image to display.

[2192] Step 2:

[2193] The server sends the selected image to the terminal.

[2194] Step 3:

[2195] The terminal displays the selected image to the user.

[2196] Handling Notification Functions

[2197] Step 1:

[2198] If the server identifies that a new image has been uploaded to the cloud storage and belongs to a particular person, it generates a notification.

[2199] Step 2:

[2200] The server sends a notification to the user terminal.

[2201] Step 3:

[2202] The device receives the notification and displays the message "New photo added" to the user.

[2203] Photo display and filtering process

[2204] Step 1:

[2205] The user accesses the photo viewing screen and enters search criteria (event name, date, etc.).

[2206] Step 2:

[2207] The terminal sends the search criteria to the server.

[2208] Step 3:

[2209] The server searches for relevant photos from the cloud storage based on the search criteria.

[2210] Step 4:

[2211] The server sends the search results to the user terminal.

[2212] Step 5:

[2213] The terminal receives the search results and displays them to the user.

[2214] Processing learning as we grow

[2215] Step 1:

[2216] The server periodically passes the accumulated image data to the AI ​​engine to prepare for retraining.

[2217] Step 2:

[2218] The server sends the AI ​​engine a new image dataset and instructs it to retrain.

[2219] Step 3:

[2220] The AI ​​engine retrains the facial recognition model using the new dataset.

[2221] Step 4:

[2222] The AI ​​engine returns the retrained model to the server.

[2223] Step 5:

[2224] The server uses the improved model for future facial recognition processing.

[2225] This processing step allows users to efficiently and accurately search for images of specific people and track their developmental progress. The addition of an emotion engine also enables personalized image display that reflects the user's emotions.

[2226] Example 2

[2227] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2228] In modern life, there is a need to efficiently manage large amounts of professionally taken images and easily search for images related to specific people or events. However, conventional systems require users to manually review and categorize images individually, which is time-consuming. Furthermore, they do not adjust image display based on the user's emotions, which hinders the quality of the user experience.

[2229] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for uploading images taken by a professional to cloud storage, means for automatically identifying a person's face from the uploaded image and extracting the facial features, and means for identifying a specific person based on the extracted features. This allows a user to automatically identify images of a specific person and display them in a list. The server also includes means for analyzing the user's emotions regarding the displayed image, means for selecting the next image to display based on the analyzed emotional data, and means for sending a notification when a new image is uploaded. This allows the image display to be adjusted according to the user's emotions, providing a more personalized user experience. Furthermore, the server includes means for continuously analyzing accumulated image data to learn the growth process of a specific person and means for searching and filtering images based on specific events and dates and times, allowing users to quickly and easily search for the images they are looking for.

[2230] "Professional" refers to an individual or organization that has specialized knowledge and skills in a particular field and performs work in that field as a profession.

[2231] "Image" means any representation of visual information in digital or analog form, including photographs, illustrations, graphics, etc.

[2232] "Cloud storage" refers to an online storage service for storing, managing, and backing up data over the Internet.

[2233] "Automatic face identification" refers to the process of detecting a human face in an image and identifying that face using a specific algorithm.

[2234] "Extracting facial features" refers to using a face detection algorithm to obtain the characteristic points and patterns of a person's face (e.g., eye position, nose shape, mouth size, etc.) as digital data.

[2235] "Identifying a specific person" refers to the process of identifying a person based on extracted facial features and comparing them with existing data in a database.

[2236] "Analyzing emotions" refers to the process by which the emotion engine identifies a user's emotional state (e.g., joy, surprise, sadness, etc.) based on data such as the user's facial expressions, tone of voice, and text input.

[2237] "Selecting the next image to display based on emotional data" refers to the process of using analyzed emotional data to automatically select and display an image that suits the user.

[2238] "Send notification" means that the system will notify the user through various methods (e.g., push notification, email, text message, etc.) when new images are uploaded to the cloud storage.

[2239] "Continuously analyzing stored image data" refers to periodically evaluating and analyzing stored image data and continuing to analyze it to identify new characteristics or changes in a particular person.

[2240] "Searching and filtering images based on specific events and dates and times" refers to the process of searching the system for relevant images based on specific criteria specified by the user (e.g., event name, date and time, etc.) and displaying only those images that match those criteria.

[2241] This invention relates to a system for efficiently managing a large number of images taken by professionals and adjusting the display according to the user's emotions. The system is composed of the following elements:

[2242] User terminal

[2243] server

[2244] Cloud Storage

[2245] AI Engine

[2246] Emotion Engine

[2247] First, the images taken by the professionals are uploaded to cloud storage. When the user creates an account using their device, the server validates the input information and stores it in the database, allowing the user to access the system.

[2248] The server then sends the images stored in cloud storage to an AI engine, which uses a face detection algorithm to identify people in the images and extract facial features. The server then compares the extracted features with an existing database to identify specific people. The identified images are added to a list and displayed on the user's device.

[2249] As a user browses images, the emotion engine analyzes the user's reactions (facial expressions, tone of voice, and text input) to identify their emotional state. The analyzed emotion data is sent to the server and used to select the next image to be displayed. For example, if a user expresses positive emotion toward a particular image, the server selects an image with a similar theme and instructs the user's device to display it. Conversely, if a negative emotion is recognized, the server selects an image with a different theme or person.

[2250] Additionally, when new images are uploaded to the cloud storage, the server sends a notification to the user's device, which includes a message that a new image has been added, allowing the user to immediately check the new image.

[2251] Other features of the system include a means to continuously analyze the stored image data to learn the developmental progression of a particular person, and the ability to search and filter images based on specific events or dates and times, allowing users to quickly and easily find the images they are looking for.

[2252] A specific example is shown below.

[2253] 1. A user accesses the application, creates an account, and submits their input information (e.g., "Yamada Taro, first grade elementary school student, 2023").

[2254] 2. The server validates the information, stores it in the database, and sends a notification to the user device that the account was successfully created.

[2255] 3. A professional photographer will upload photos from the April 2023 entrance ceremony to cloud storage.

[2256] 4. The server stores these photos in cloud storage and assigns metadata to each photo.

[2257] 5. The server retrieves the photo from the cloud storage and sends it to the AI ​​engine.

[2258] 6. The AI ​​engine identifies the face, extracts features, and compares them with a database to identify a specific person (e.g., "Taro Yamada").

[2259] 7. The emotion engine analyzes the user's reaction and sends the emotion data to the server.

[2260] 8. The server selects the next image to display based on the emotion data and displays it on the user's device.

[2261] 9. When a new image is uploaded, the server sends a notification and the user device displays it.

[2262] 10. The user sets search filters by event or date and time, and the server provides the corresponding images.

[2263] The following is an example of a prompt:

[2264] "Search for and display photos of Taro Yamada's 2023 entrance ceremony."

[2265] The system optimizes the user experience, allowing parents to easily browse photos from important events, and uses an emotion engine to display personalized images based on the user's emotions.

[2266] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2267] Step 1:

[2268] A user accesses an application and creates an account. The specific actions of this step are that the user enters their name, email address, password, and other required information, and clicks the submit button. The entered data (name, email address, password) is sent to the server. The server receives it and performs validation. If validation is successful, the server saves the data in the database and sends a notification to the user terminal that the account was created successfully. The user terminal displays the message "Account created." The input is the user's information, and the output is a notification that the account was created successfully.

[2269] Step 2:

[2270] The professional photographer uploads the photos he has taken to cloud storage. The specific operations of this step are that the professional photographer selects the image files taken for each event and uploads them to a cloud storage service (e.g., Amazon S3). The uploaded images are assigned metadata such as the event name, date, and location. This information is sent to a server. The server receives this information and stores it in cloud storage. The input is the image files and metadata, and the output is the images stored in cloud storage.

[2271] Step 3:

[2272] The server sends the image stored in cloud storage to the AI ​​engine. The specific operation of this step is that the server retrieves the image file from cloud storage and sends it to the AI ​​engine (e.g., Google Cloud Vision API). The input is the image file retrieved from cloud storage, and the output is the face recognition result by the AI ​​engine. The AI ​​engine uses a face detection algorithm to automatically identify the face in the image and extract facial features (e.g., eye position, nose shape, mouth size, etc.).

[2273] Step 4:

[2274] The server receives the feature data sent from the AI ​​engine and compares it with an existing database. The specific operation of this step is that the server compares the feature data with an existing person database and identifies a specific person based on the degree of match. The input is the feature data obtained from the AI ​​engine, and the output is the result of identifying a specific person. The image of the identified person is added to a list and displayed on the user's device.

[2275] Step 5:

[2276] The user device displays the image and sends the user's reaction to the emotion engine. Specific operations of this step include the user device displaying the identified image and sending the user's facial expression, tone of voice, and text input to the emotion engine (e.g., Microsoft Azure Emotion API). The input is the user's reaction data, and the outp...

Claims

1. A means to upload professionally taken images to cloud storage; means for automatically identifying human faces and extracting facial features from uploaded images; A means for identifying a particular person based on the extracted features; A means for displaying a list of images of the identified specific person; A way to be notified when new images are uploaded, A system including:

2. 10. The system of claim 1, further comprising means for continuously analyzing the stored image data to learn the developmental progression of a particular person.

3. The system of claim 1 further comprising means for searching and filtering images based on specific events and dates.

Images