System

Abstract

A system is provided.SOLUTION: A system comprising means for taking or scanning an old monochrome photograph, means for transmitting the taken or scanned monochrome photograph data to a server, means for performing image correction on the server to convert the monochrome photograph into color, means for correcting the focus of the colorized photograph to make it sharp, means for interacting with a user using a generative AI to customize the photograph based on the user's instructions, and means for finally providing the customized photograph to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention aims to solve the problem of colorizing old monochrome photographs and making out-of-focus photographs clearer. Conventional technologies for colorizing monochrome photographs and correcting focus require specialized knowledge and expensive costs, making them inaccessible to average users. Furthermore, the means for customizing photographs are limited, making it difficult for users to edit photos as desired. There has been a demand, particularly for those in their 50s and older, for an easy-to-use, low-cost way to restore memorable photographs. [Means for solving the problem]

[0005] The present invention solves the problem with a system that includes a means for taking or scanning old monochrome photographs, a means for transmitting the taken or scanned monochrome photograph data to a server, a means for performing image correction on the server to convert the monochrome photograph to color, a means for correcting the focus of the colorized photograph to make it clearer, a means for interacting with the user using a generative AI to customize the photograph based on the user's instructions, and a means for finally providing the customized photograph to the user. This allows users to easily and at low cost convert old monochrome photographs into clear color photographs, and further allows them to make desired modifications through interaction with the generative AI.

[0006] An "old black and white photograph" is a black and white photograph taken in the past, and is an image that does not contain color information.

[0007] "Photographing or scanning means" refers to a device or method that converts a physical photograph into digital data using a camera or scanner.

[0008] "Monochrome photograph data" refers to a digitized image file of a photographed or scanned monochrome photograph.

[0009] A "server" is a high-performance computer that provides services and processes data over a network.

[0010] "Image correction" refers to the process of improving the image quality of a digital image by adjusting color, brightness, focus, etc.

[0011] "Means for converting to color" refers to an algorithm or technology that applies color to monochrome photographic data and converts it into a color image.

[0012] "Focus correction" is the process of detecting blurred parts of an image and sharpening them.

[0013] "Generative AI" refers to artificial intelligence that automatically customizes and edits photos based on human instructions.

[0014] "Interacting with the user" refers to the process in which the generative AI responds to input and questions from the user and receives instructions.

[0015] "Customization" refers to the process of making specific modifications to an image, such as adjusting the color or removing the background, according to the user's preferences.

[0016] A "final customized photograph" is a digital image that has undergone final retouching and is finished to the user's specifications.

[0017] "Means of providing" refers to the method of providing the final photo data to the user, such as via a download link. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The following describes in detail an embodiment of the present invention. The present invention is a system for converting old black-and-white photographs to color and sharpening out-of-focus photographs. This system is implemented through a dedicated app that is easy for users to operate, and includes the following processing flow:

[0040] Taking and uploading photos

[0041] 1. The user prepares an old black-and-white photo and launches the dedicated app.

[0042] 2. The device (smartphone or tablet) starts the camera and the user takes a photo. A shooting guide is displayed, instructing the user on the appropriate shooting angle and lighting conditions.

[0043] 3. The user can preview the photo they have taken and retake it if necessary.

[0044] 4. The user selects the photo they are satisfied with and presses the "Upload" button in the dedicated app.

[0045] 5. The device compresses the selected photo data and sends it to the server.

[0046] AI-based initial processing

[0047] 1. The server receives the photo data and runs the image correction AI. First, it converts the monochrome photo to color. This process involves an algorithm that colors each pixel to create natural-looking tones.

[0048] 2. A focus correction algorithm is then used to detect blurry areas in the colorized photo and a sharpening filter is applied, resulting in a clearer image.

[0049] Customization Process

[0050] 1. The server uses the generative AI model to generate a dialogue interface with the user, through which the user can provide specific customization instructions.

[0051] 2. Users can interact with the AI ​​to adjust the colors to their liking or remove unnecessary background elements. For example, they can communicate specific requests such as "I want the background blurred" or "I want the costume color changed to blue."

[0052] 3. The generative AI model analyzes the user's instructions and applies real-time corrections to the initially processed photo.

[0053] 4. The server displays a preview of the edited result to the user and asks for confirmation. The user can make further adjustments while viewing the preview.

[0054] Final data storage and distribution

[0055] 1. The server generates the final edited photo data and creates a permanent link.

[0056] 2. The server saves the final photo data and provides the user with a download link via email or in-app notification.

[0057] 3. The user clicks on the provided link and downloads the final photo data via a dedicated app or browser.

[0058] 4. Users can save the downloaded photos to their own devices and print them out or share them with family and friends as needed.

[0059] Specific examples

[0060] Example 1: Colorizing a black and white wedding photo

[0061] 1. The user launches the app and takes an old black-and-white wedding photo with their camera.

[0062] 2. The device sends the captured image data to the server.

[0063] 3. The server colorizes and sharpens the received photo.

[0064] 4. The server interacts with the user through the generated AI model and receives instructions on adjusting the background.

[0065] 5. The server generates the final retouched photo and provides the user with a download link.

[0066] 6. User downloads and saves the final photo via the link.

[0067] Example 2: Sharpening an out-of-focus photo of grandparents

[0068] 1. The user digitizes old photos using a scanner and uploads them using a dedicated app.

[0069] 2. The device sends the scanned data to the server.

[0070] 3. The server receives the data and performs focus correction and colorization processing.

[0071] 4. The server interacts with the generated AI model and modifies facial details based on the user's wishes.

[0072] 5. The server generates the final photo after user confirmation and provides a download link.

[0073] 6. Users download, save and share.

[0074] By implementing this invention, old black-and-white photographs can be easily colorized and out-of-focus photographs can be sharpened, allowing users to vividly relive past memories. This makes the system extremely useful, especially for people in their 50s and older.

[0075] The processing flow will be explained below.

[0076] Step 1:

[0077] The user prepares an old black-and-white photo and launches the dedicated app.

[0078] Step 2:

[0079] The device (smartphone or tablet) starts the camera and the user takes a photo. A shooting guide is displayed, instructing the appropriate shooting angle and lighting conditions.

[0080] Step 3:

[0081] The user can preview the photo they have taken and take another photo if necessary.

[0082] Step 4:

[0083] The user selects the photo they are satisfied with and presses the "Upload" button in the dedicated app.

[0084] Step 5:

[0085] The terminal compresses the selected photo data and transmits it to the server.

[0086] Step 6:

[0087] The server receives the photo data and temporarily stores it in an input database, where it checks the data for consistency and for missing or error information.

[0088] Step 7:

[0089] The server runs an image correction AI to convert the received monochrome photo into color, which then colors each pixel to produce a natural color tone.

[0090] Step 8:

[0091] The server passes the colorized photo to a focus correction algorithm, which detects blurry areas and applies a sharpening filter.

[0092] Step 9:

[0093] The server sends the color photo data after initial processing to the generative AI model.

[0094] Step 10:

[0095] The server generates an interactive interface with the user through the generated AI model.

[0096] Step 11:

[0097] The user can use the dialogue interface to make color adjustments or remove unnecessary backgrounds, for example, by inputting instructions such as "I want the background to be blurred" or "I want the color of the costumes to be changed to blue."

[0098] Step 12:

[0099] The generative AI model analyzes the user's instructions and applies corrections to the initially processed photo.

[0100] Step 13:

[0101] The server displays the correction results in real time and asks the user for confirmation, allowing the user to give further instructions while viewing the preview and adjust the final result.

[0102] Step 14:

[0103] The server generates the final edited photo data and creates a permanent link.

[0104] Step 15:

[0105] The server stores the final photo data and provides the user with a download link via email or in-app notification.

[0106] Step 16:

[0107] The user clicks on the provided link and downloads the final photo data via a dedicated app or browser.

[0108] Step 17:

[0109] Users can save the downloaded photos to their own devices and print them out or share them with family and friends as needed.

[0110] Example 1

[0111] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0112] Conventional photo correction systems often require manual work, whether it's converting old black-and-white photos to color or correcting out-of-focus photos to make them clearer, making the process extremely time-consuming. Furthermore, specific corrections and customizations require users to have advanced skills, making them difficult for average users without specialized knowledge. For this reason, there is a demand for a system that allows users to easily perform high-quality photo corrections.

[0113] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0114] In this invention, the server includes a device that takes or scans old black-and-white photographs, a device that transmits the taken or scanned black-and-white photograph data to the server, a program that performs image correction on the server and converts the black-and-white photographs to color, a program that corrects the focus of the colorized photographs to make them clearer, a program that interacts with the user using a generative AI model and customizes the photographs based on the user's instructions, and a device that finally provides the customized photographs to the user. This allows users to easily perform high-quality photo correction without specialized knowledge.

[0115] An "old black and white photograph" is a black and white (monochrome) photograph taken in the past, and refers to an image that does not contain color information.

[0116] A "photographing or scanning device" is a device for converting a physical photograph into a digital image, and includes hardware devices such as a camera or scanner.

[0117] "Transmitting device" refers to a hardware or software device for transferring photographed or scanned digital image data to a server via a network.

[0118] "Server" refers to a remote computer or cloud-based computing system for receiving and processing digital image data.

[0119] "Image correction program" refers to a software application that performs operations such as adjusting color tone and correcting exposure on digital images.

[0120] A "color conversion program" refers to a software algorithm that adds color information to each pixel of a monochrome image to convert it into a natural color image.

[0121] A "focus and sharpening program" refers to software that detects blurry areas in an image and applies a sharpening filter to improve the image.

[0122] "Generative AI model" refers to an artificial intelligence model that interacts with users, analyzes their instructions based on natural language prompts, and is responsible for making corrections and improvements to images.

[0123] "Customization program" refers to software that modifies images based on a user's specific needs.

[0124] The "providing device" refers to a device for distributing the final edited image data to the user, and includes email, cloud storage services, etc.

[0125] The system of this invention is mainly designed to convert old black and white photos into color and sharpen out-of-focus photos, allowing users to vividly relive their past memories. The whole system consists of three main components: the user, the terminal, and the server.

[0126] First, the user prepares an old black-and-white photograph and launches the dedicated application. This application runs on devices such as smartphones and tablets, and activates the camera to take a photo. Shooting guides are displayed, allowing the user to take a photo with the optimal lighting and angle.

[0127] The device then displays a preview of the captured photo, allowing the user to review and retake the photo until satisfied. Once a satisfactory photo is selected, the device compresses the photo data and sends it to the server.

[0128] The server receives the photo data and performs image correction. The image correction process first converts the monochrome photo to color. Specifically, it uses Python's OpenCV library to apply an algorithm that estimates the appropriate color for each pixel and generates natural-looking tones. Next, a focus correction algorithm using TENSORFLOW® is applied, detecting blurred areas and applying a sharpening filter to make the entire photo clearer.

[0129] Additionally, a customization process is provided in which a generative AI model interacts with the user. This generative AI model uses advanced natural language processing techniques, such as GPT-4 (registered trademark), to modify photos based on prompts entered by the user. Through the dialogue interface, users can provide specific customization instructions, such as "blur the background" or "change the color of the outfit to blue." The generative AI model analyzes these instructions and applies them to the photo in real time.

[0130] The final customized photo is generated as high-resolution image data by the server. The server stores this data in cloud storage and generates a permanent link that is provided to the user. The user receives a download link via email or in-app notification, and can download the final photo data through a dedicated application or browser and save it to their device. This allows the user to print out the final photo or share it with family and friends.

[0131] As a concrete example, we will show the process of colorizing a black-and-white wedding photo. The user launches a dedicated app and takes a photo of an old black-and-white wedding photo with their camera. The device sends the captured data to a server, which then colorizes and sharpens the received photo. The background is then adjusted based on the user's instructions via a generative AI model, and the user is provided with a download link for the final edited photo.

[0132] By implementing this invention, users can easily colorize old monochrome photographs and sharpen out-of-focus photographs without any specialized knowledge. This provides a very useful system for vividly reviving past memories, especially for those in their 50s and older.

[0133] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0134] Step 1:

[0135] The user prepares an old black-and-white photo and launches the dedicated app.

[0136] Input: Physical black and white photo

[0137] Output: Launching the dedicated app

[0138] Specific operation: The user prepares a physical black-and-white photo and launches a dedicated app on their smartphone or tablet.

[0139] Step 2:

[0140] The device will activate the camera and take a photo.

[0141] Input: User-initiated camera activation and shooting instructions

[0142] Output: Digitized black and white photo capture

[0143] Specific operation: The device's camera is activated, a shooting guide is displayed, and the user presses the shooting button to digitize a monochrome photo.

[0144] Step 3:

[0145] The device displays a preview of the photo for the user to review and retake.

[0146] Input: Digitized black and white photograph

[0147] Output: Show preview image

[0148] Specific operation: The device displays a preview of the photo taken and displays the "Confirm" and "Retake" buttons. The user checks the preview and presses the "Retake" button if they need to take another photo.

[0149] Step 4:

[0150] The user selects a photo they are satisfied with and presses the upload button.

[0151] Input: A preview image selected by the user.

[0152] Output: Ready to upload status

[0153] Specific operation: The user presses the "Confirm" button, which takes them to the upload screen of the dedicated app. The user presses the "Upload" button.

[0154] Step 5:

[0155] The terminal compresses the selected photo data and transmits it to the server.

[0156] Input: A digitized black and white photograph selected by the user

[0157] Output: Compressed photo data sent to the server

[0158] Specific operation: The device compresses the image data and sends it to a server via the Internet.

[0159] Step 6:

[0160] The server receives the photo data and performs image correction.

[0161] Input: Compressed photo data

[0162] Output: Colorized and focused image

[0163] What it does: The server receives the photo data and colorizes the monochrome photo using Python's OpenCV library, then uses TensorFlow to apply a focus correction algorithm to make the photo clearer.

[0164] Step 7:

[0165] The server uses the generative AI model to generate a dialogue interface with the user.

[0166] Input: Colorized and focus-corrected image data

[0167] Output: Generate a conversational interface

[0168] Specific operation: The server uses a generative AI model (e.g., GPT-4) to generate a dialogue interface and provides a screen for the user to input customization instructions.

[0169] Step 8:

[0170] The user gives specific customization instructions.

[0171] Input: Prompt text given by the user (e.g., "I want the background to be blurred" or "I want the color of the costume to be blue")

[0172] Output: Instruction analysis results by the generative AI model

[0173] Specific operation: The user inputs a prompt sentence through the dialogue interface, and the AI ​​receives and analyzes it.

[0174] Step 9:

[0175] A generative AI model modifies photos based on user instructions.

[0176] Input: Prompt statement analysis result

[0177] Output: Corrected image data

[0178] How it works: The generative AI model modifies specific parts of the image based on user instructions and makes corresponding adjustments in real time.

[0179] Step 10:

[0180] The server displays a preview of the correction results to the user.

[0181] Input: Corrected image data

[0182] Output: Preview screen display

[0183] What happens: The server previews the corrected image to the user, allowing the user to make further adjustments.

[0184] Step 11:

[0185] The server generates the final edited photo data and creates a fixed link.

[0186] Input: Final corrected image data

[0187] Output: Cloud storage of image data and generation of a permanent link

[0188] Specific operation: The server generates a high-resolution image after correction is completed, saves it to cloud storage, and then creates a permanent link.

[0189] Step 12:

[0190] The server notifies the link and provides it to the user.

[0191] Input:permalink

[0192] Output: User notification (email or in-app notification)

[0193] Specific operation: The server notifies the user of the fixed link and provides a download link.

[0194] Step 13:

[0195] The user clicks on the provided link to download the final photo data.

[0196] Input:permalink

[0197] Output: Final downloaded photo data

[0198] Specific operation: The user clicks on the link and downloads the final photo data via a dedicated app or web browser.

[0199] Step 14:

[0200] Save the photos you download to your device and share them as needed.

[0201] Input: Downloaded photo data

[0202] Output: Photos stored on the device and shared photos

[0203] What it does: Users can save the downloaded photos to their devices, share them with family and friends, and even print them out.

[0204] (Application example 1)

[0205] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0206] In recent years, advances in digital technology have led to an increasing number of users wanting to digitize and preserve old photographs. However, converting old black-and-white photographs to color and sharpening blurry photographs is a time-consuming process that requires specialized knowledge. Furthermore, there are limited means to easily provide such services, especially in brick-and-mortar stores. This makes it difficult for customers to restore photos at their own convenience, leading to lower service satisfaction. Therefore, there is a need for a system that can easily colorize and sharpen old photographs in-store.

[0207] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0208] In this invention, the server includes means for taking or scanning old monochrome photographs, means for transmitting the taken or scanned monochrome photograph data to the server, means for performing image correction on the server to convert the monochrome photograph to color, means for correcting the focus of the colorized photograph to make it clearer, means for interacting with the user using a generative AI and customizing the photograph based on the user's instructions, means for providing the user with the final customized photograph, and means for the user to use the application that performs the photo customization process in a physical store environment. This allows customers to easily colorize and sharpen old photographs in a physical store.

[0209] An "old black-and-white photograph" is a photograph taken in the past that lacks color and is expressed in black and white rather than color.

[0210] "Means of photographing or scanning" refers to the use of a device such as a camera or scanner to obtain a photograph as digital data.

[0211] A "transmitting means" is a communication means for sending digital data over the Internet or a local network to a remote server.

[0212] A "server" is a computer system that receives and processes digital data.

[0213] "Image correction means" refers to software or algorithms for adjusting contrast and correcting color of digital images.

[0214] "Methods for converting to color" are techniques for adding color to monochrome photographs to reproduce them as natural color photographs.

[0215] "Focus correction and sharpening" is a method for sharpening blurry photos and improving the clarity of details.

[0216] "Generative AI" is a system that uses an artificial intelligence model to interact with users in natural language and process images based on their explanations.

[0217] "Means for interacting with users and customizing" refers to technology that uses AI to understand users' requests and modify or process specific parts of images.

[0218] A "means for providing a customized photograph to a user" is a method by which the final, enhanced digital image can be downloaded or viewed by the user.

[0219] "Methods available to users in a physical store environment" refers to the use of applications and digital devices by customers to receive services within a physical store.

[0220] The following describes in detail the mode for carrying out the invention. This system has various functions for converting old black-and-white photos to color and sharpening out-of-focus photos. It is intended for use in brick-and-mortar stores, and customers can easily use these functions using a dedicated application.

[0221] Hardware and software used

[0222] The system uses the following hardware and software:

[0223] Hardware:

[0224] Devices (smartphones, tablets, cameras)

[0225] Server (high performance processor, internet connection)

[0226] software:

[0227] OpenCV (image capture, image processing)

[0228] PIL (Python Imaging Library) (Image Processing)

[0229] requests (HTTP communication library)

[0230] Generative AI model (user interaction, image customization)

[0231] Processing flow explanation

[0232] Taking and uploading photos

[0233] The user launches a dedicated app and prepares an old black-and-white photo. The device then activates the camera, and the user takes a photo. A shooting guide is displayed, instructing on the appropriate shooting angle and lighting conditions. The user can check a preview of the photo they have taken and take another photo if necessary. When the user selects a photo they are satisfied with and presses the upload button, the device compresses the photo data and sends it to the server.

[0234] AI-based initial processing

[0235] The server receives the photo data and runs an image correction AI, first converting the black and white photo to color, then using a focus correction algorithm to detect blurry areas in the colorized photo and applying a sharpening filter.

[0236] Customization Process

[0237] The server uses the generative AI model to generate a dialogue interface for the user. The user can use this interface to give specific customization instructions. For example, they can express specific requests such as "I want the background blurred" or "I want the costume color changed to blue." The generative AI model analyzes the user's instructions and applies real-time corrections to the initially processed photo. The server then displays a preview of the corrections to the user and receives confirmation and additional instructions.

[0238] Final data storage and distribution

[0239] The server generates the final edited photo data and creates a permanent link. The server saves the final photo data and provides the user with a download link via email or in-app notification. The user clicks the link and downloads the final photo data via a dedicated app or browser. The user saves the downloaded photo to their device and can print it out or share it with family and friends as needed.

[0240] Examples of concrete examples and prompts

[0241] As a concrete example, customers can take old family photos with the app in a physical store where their families gather, and have them colorized and sharpened on the spot. This service allows customers to vividly relive their past memories and share them as new memories.

[0242] Example prompt sentence:

[0243] 1. Take a photo or upload an existing one.

[0244] 2. Sending photos to server... Please wait.

[0245] 3. Processing the photo… We will correct the focus and add color to the original photo.

[0246] 4. Processing completed! Please check.

[0247] 5. Do you want to customize further? For example, "I want the background to be blurred" or "I want the costume color to be changed to blue."

[0248] This system allows customers to easily colorize and sharpen old photos in-store.

[0249] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0250] Step 1:

[0251] Photographing and scanning

[0252] The user prepares an old black-and-white photo and launches a dedicated app. The device launches the camera and the user takes a photo. A shooting guide is displayed, instructing on the appropriate shooting angle and lighting conditions. The input is the old black-and-white photo, and the output is a digital copy of the photo. A preview is provided to confirm that the appropriate shooting conditions are met, allowing the user to decide whether or not to retake the photo.

[0253] Step 2:

[0254] Upload

[0255] When the user selects a photo they are satisfied with and presses the upload button, the device compresses the photo data and sends it to the server. The input is a digital photo, and the output is compressed photo data. This process improves data transfer speed and reduces server load.

[0256] Step 3:

[0257] Initial Data Processing

[0258] The server receives the photo data and first runs the image correction AI. The input is compressed photo data, and the output is colorized photo data. Specifically, it applies an algorithm that adds color to each pixel of the monochrome photo and generates natural-looking tones.

[0259] Step 4:

[0260] Focus Correction

[0261] The server then uses a focus correction algorithm to detect blurred areas in the colorized photo and applies a sharpening filter. The input is the colorized photo data, and the output is the sharpened photo data. Specifically, the server runs a filter to improve the contrast and detail in the blurred areas.

[0262] Step 5:

[0263] Customization with generative AI

[0264] The server uses the generative AI model to generate a dialogue interface for the user. The user can then provide specific customization instructions through this interface. The input is the user's text-based instructions, and the output is customized photo data. Specific operations include analyzing requests such as "blur the background" or "change the color of the costume to blue," and reflecting the results in image processing.

[0265] Step 6:

[0266] Preview and fix

[0267] The server displays a preview of the correction results to the user and receives confirmation and additional instructions from the user. The input is customized photo data, and the output is the final corrected photo data. Specifically, it provides an interface that allows the user to make further adjustments while viewing the preview.

[0268] Step 7:

[0269] Final data storage and distribution

[0270] The server generates the final corrected photo data and creates a permanent link. The server saves the final photo data and provides the user with a download link via email or in-app notification. The input is the final corrected photo data and the output is a permanent link. Specifically, this involves saving the photo in a database and generating a link to provide it to the user.

[0271] Through this process, customers can easily colorize and sharpen old photos in-store.

[0272] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0273] The following describes in detail the embodiments of the present invention. The present invention combines a system for colorizing old black-and-white photos and sharpening out-of-focus photos with an emotion engine that recognizes the user's emotions. This system is implemented through a dedicated app that is easy for users to operate, and includes the following processing flow:

[0274] Taking and uploading photos

[0275] 1. The user prepares an old black-and-white photo and launches the dedicated app.

[0276] 2. The device (smartphone or tablet) starts the camera and the user takes a photo. A shooting guide is displayed, instructing the user on the appropriate shooting angle and lighting conditions.

[0277] 3. The user can preview the photo they have taken and retake it if necessary.

[0278] 4. The user selects the photo they are satisfied with and presses the "Upload" button in the dedicated app.

[0279] 5. The device compresses the selected photo data and sends it to the server.

[0280] AI-based initial processing

[0281] 1. The server receives the photo data and temporarily stores it in an input database, where it checks the data for consistency and omissions or errors.

[0282] 2. The server runs the image correction AI to convert the received monochrome photo to color. The image correction AI colors each pixel to produce natural-looking colors.

[0283] 3. The server passes the colorized photo to a focus correction algorithm, which detects blurry areas and applies a sharpening filter.

[0284] Emotion engine processing

[0285] 1. The server activates the emotion engine and uses the camera and microphone to analyze the user's voice and facial expressions.

[0286] 2. The device transmits the user's voice and facial expression data to the emotion engine in real time.

[0287] 3. The emotion engine analyzes the received data and determines the user's emotion. For example, if the user is smiling, it is determined to be "joy," and if they are frowning, it is determined to be "confusion."

[0288] 4. The server receives the judgment results from the emotion engine and works with the generation AI.

[0289] Customization Process

[0290] 1. The server uses the generative AI model to generate a dialogue interface with the user.

[0291] 2. The user enters specific customization instructions through the dialogue interface, such as "I want the background to be blurred" or "I want the color of the costume to be blue."

[0292] 3. The generative AI model analyzes the user's instructions and applies corrections to the initially processed photo.

[0293] 4. The emotion engine analyzes the user's emotions in real time and provides feedback to the generative AI, allowing the generative AI model to suggest additional modifications based on the emotions.

[0294] 5. The server displays the edited results in real time and asks the user for confirmation. The user can make further adjustments while viewing the preview.

[0295] Final data storage and distribution

[0296] 1. The server generates the final edited photo data and creates a permanent link.

[0297] 2. The server saves the final photo data and provides the user with a download link via email or in-app notification.

[0298] 3. The user clicks on the provided link and downloads the final photo data via a dedicated app or browser.

[0299] 4. Users can save the downloaded photos to their own devices and print them out or share them with family and friends as needed.

[0300] Specific examples

[0301] Example 1: Colorization of Black and White Wedding Photos and Emotion-Based Customization

[0302] 1. The user launches the app and takes an old black-and-white wedding photo with their camera.

[0303] 2. The device sends the captured image data to the server.

[0304] 3. The server colorizes and sharpens the received photo.

[0305] 4. The server interacts with the user through the generative AI model and emotion engine and receives instructions on adjusting the background.

[0306] 5. The emotion engine detects the user's happiness and suggests warmer background colors.

[0307] 6. The server generates the final retouched photo and provides the user with a download link.

[0308] 7. User downloads and saves the final photo via the link.

[0309] Example 2: Sharpening out-of-focus photos of grandparents and emotion-based customization

[0310] 1. The user digitizes old photos using a scanner and uploads them using a dedicated app.

[0311] 2. The device sends the scanned data to the server.

[0312] 3. The server receives the data and performs focus correction and colorization processing.

[0313] 4. The server interacts with the user through the generative AI model and emotion engine, and performs facial detail correction.

[0314] 5. The emotion engine analyzes the user's emotions, and if it detects "confusion," the generative AI will suggest more detailed adjustment methods.

[0315] 6. The server generates the final photo after user confirmation and provides a download link.

[0316] 7. Users can download, save, or share.

[0317] By implementing this invention, not only can old black-and-white photographs be easily colorized and out-of-focus photographs be made clearer, but the system can also be optimally customized based on the user's emotions, allowing users to relive their past memories in a richer way. This makes the system extremely useful, especially for those in their 50s and older.

[0318] The processing flow will be explained below.

[0319] Step 1:

[0320] The user prepares an old black-and-white photo and launches the dedicated app.

[0321] Step 2:

[0322] The device (smartphone or tablet) starts the camera and the user takes a photo. A shooting guide is displayed, instructing the appropriate shooting angle and lighting conditions.

[0323] Step 3:

[0324] The user can preview the photo they have taken and take another photo if necessary.

[0325] Step 4:

[0326] The user selects the photo they are satisfied with and presses the "Upload" button in the dedicated app.

[0327] Step 5:

[0328] The terminal compresses the selected photo data and transmits it to the server.

[0329] Step 6:

[0330] The server receives the photo data and temporarily stores it in an input database, where it checks the data for consistency and for missing or error information.

[0331] Step 7:

[0332] The server runs an image correction AI to convert the received monochrome photo into color, which then colors each pixel to produce a natural color tone.

[0333] Step 8:

[0334] The server passes the colorized photo to a focus correction algorithm, which detects blurry areas and applies a sharpening filter.

[0335] Step 9:

[0336] The server runs the emotion engine and uses the camera and microphone to analyze the user's voice and facial expressions.

[0337] Step 10:

[0338] The device transmits the user's voice and facial expression data to the emotion engine in real time.

[0339] Step 11:

[0340] The emotion engine analyzes the received data and determines the user's emotion. For example, if the user is smiling, it is determined to be "joy," and if they are frowning, it is determined to be "confusion."

[0341] Step 12:

[0342] The server receives the judgment results from the emotion engine and works in conjunction with the generation AI.

[0343] Step 13:

[0344] The server uses the generative AI model to generate a dialogue interface with the user.

[0345] Step 14:

[0346] The user enters specific customization instructions through a dialogue interface, such as "I want the background to be blurred" or "I want the color of the costume to be changed to blue."

[0347] Step 15:

[0348] The generative AI model analyzes the user's instructions and applies corrections to the initially processed photo.

[0349] Step 16:

[0350] The emotion engine analyzes the user's emotions in real time and provides feedback to the generative AI.

[0351] Step 17:

[0352] The generative AI model suggests additional modifications based on the user's sentiment.

[0353] Step 18:

[0354] The server displays the correction results in real time and asks the user for confirmation, allowing the user to make further adjustments while viewing the preview.

[0355] Step 19:

[0356] The server generates the final edited photo data and creates a permanent link.

[0357] Step 20:

[0358] The server stores the final photo data and provides the user with a download link via email or in-app notification.

[0359] Step 21:

[0360] The user clicks on the provided link and downloads the final photo data via a dedicated app or browser.

[0361] Step 22:

[0362] Users can save the downloaded photos to their own devices and print them out or share them with family and friends as needed.

[0363] Example 2

[0364] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0365] Conventional photo restoration systems provide methods for colorizing old monochrome photographs and correcting focus, but do not offer customization that takes into account the user's emotions. As a result, it takes a lot of time and effort to achieve results that satisfy the user. The purpose of this invention is to solve these problems and provide a system that is easy for users to operate and that allows customization based on emotions.

[0366] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0367] In this invention, the server includes means for taking or scanning old monochrome photographs, means for transmitting the taken or scanned monochrome photograph data to an information processing device, means for performing image correction on the information processing device to convert the monochrome photograph to color, means for correcting the focus of the colorized photograph to make it clearer, means for analyzing the user's voice and facial expression in real time to determine their emotions, means for interacting with the user using a generative AI and customizing the photo based on the user's instructions and the determined emotions, and means for finally providing the customized photo to the user. This enables photo restoration and customization that satisfies the user to be performed quickly and easily.

[0368] "Old monochrome photographs" are black and white photographic data that were taken in the past and do not contain color information.

[0369] "Photographing or scanning means" refers to methods and devices that use cameras or scanners to convert physical photographs or images into digital data.

[0370] "Information processing device" refers to a computer system such as a server or cloud computing system for processing captured or scanned image data.

[0371] "Method of image correction and converting black-and-white photos to color" refers to an image processing technology that assigns an appropriate color to each pixel of a black-and-white photo to reproduce natural color tones.

[0372] "Method for correcting focus and sharpening colorized photographs" refers to a technique that detects blurry areas of an image and applies a sharpening filter to them to increase the overall clarity of the image.

[0373] "Means for analyzing a user's voice and facial expressions in real time to determine emotions" refers to a system that uses a microphone or camera to collect a user's voice and facial expressions, and analyzes that data to identify the user's emotional state.

[0374] "Generative AI" refers to an artificial intelligence system that interacts with users based on input data and instructions and automatically performs the necessary customizations.

[0375] "Means for interacting with a user and customizing a photo based on the user's instructions and determined emotions" refers to a method and apparatus that receives instructions from a user via a user interface and further processes or modifies a photo taking into account the user's emotional state.

[0376] "Means for providing the final customized photo to the user" means a system that provides the final processed photo data to the user via a download link or in-app notification.

[0377] This system aims to restore a clear image by colorizing old monochrome photographs and adjusting the focus. It also analyzes the user's voice and facial expressions to determine their emotions, and customizes the photo restoration process based on those emotions to ensure the user is satisfied with the results. This system is implemented using the following hardware and software:

[0378] Hardware configuration:

[0379] Device: A device such as a smartphone, tablet, or computer that contains a camera and microphone and collects user input and emotional data.

[0380] Server: A server with high-performance processing power, where image processing, database management, and AI model execution take place.

[0381] Software configuration:

[0382] Image Enhancement Software: This includes AI algorithms that colorize old black-and-white photos and correct focus. Specific examples of the software used include OpenCV and TensorFlow.

[0383] Emotion engine: Software that analyzes the user's voice and facial expressions to determine their emotions. Specific software used includes "DeepFace" and "Google (registered trademark) Cloud Speech-to-Text."

[0384] Generative AI models: Personalize photos based on user input and emotions, using natural language processing and generative models such as GPT-3 (registered trademark) and DALL·E.

[0385] Examples:

[0386] 1. Colorization of monochrome wedding photos and emotion-based customization

[0387] The user launches the dedicated app and takes an old black-and-white wedding photo with their camera.

[0388] The device sends the captured image data to the server.

[0389] The server then colorizes and sharpens the received photographs.

[0390] The server interacts with the user through the generative AI model and emotion engine, and receives instructions for adjusting the background.

[0391] The emotion engine detects the user's happiness and suggests warmer background colors.

[0392] The server generates the final retouched photo and provides the user with a download link.

[0393] The user downloads and saves the final photo via a link.

[0394] 2. Sharpening out-of-focus photos of grandparents and emotion-based customization

[0395] Users digitize old photos using a scanner and upload them using a dedicated app.

[0396] The device sends the scanned data to the server.

[0397] The server receives the data and performs focus correction and colorization processing.

[0398] The server interacts with the user through the generative AI model and emotion engine, and performs facial detail correction.

[0399] The emotion engine analyzes the user's emotions, and if it detects "confusion," the generative AI will suggest more detailed adjustment methods.

[0400] The server generates the final photo after user confirmation and provides a download link.

[0401] Users can download, save and share.

[0402] Example prompt sentence:

[0403] 1. "Blur the background in this photo to make the person stand out."

[0404] 2. "Please change the color of the outfit in this photo to blue to balance the overall look."

[0405] 3. "I'd like to see a bit more sharpness in this photo to bring out the details."

[0406] This system allows users to easily colorize old black-and-white photos, improve their clarity, and even customize them individually, allowing users to more vividly relive their rich memories.

[0407] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0408] Step 1:

[0409] The user prepares an old black-and-white photo and launches the dedicated app.

[0410] Input: Old black and white photo

[0411] Output: Ready to be photographed or scanned

[0412] Step 2:

[0413] When the user presses the "Take Photo" button on the dedicated app, the device's camera will launch and a photo guide will be displayed.

[0414] Input: User action (pressing the "Take Photo" button)

[0415] Output: Starts the camera and displays the shooting guide

[0416] What it does: The shooting guide displays messages instructing you on the appropriate shooting angles and lighting conditions.

[0417] Step 3:

[0418] The user takes a photo using the camera and then sees a preview screen after taking the photo, which allows them to choose whether to retake the photo or use the one they just took.

[0419] Input: Photo data taken

[0420] Output: Preview screen and user choice (retake or use)

[0421] Step 4:

[0422] The user selects a photo they are satisfied with and presses the "Upload" button in the dedicated app.

[0423] Input: User operation (pressing the "Upload" button)

[0424] Output: Photo ready to upload

[0425] Step 5:

[0426] The device compresses the selected photo data and sends it to the server. The device uses a compression algorithm to minimize the data size and reduce transmission time.

[0427] Input: Photo data taken

[0428] Output: Compressed photo data, sent to server

[0429] Step 6:

[0430] The server receives the photo data and temporarily stores it in an input database, where it checks the data for consistency and omissions or errors.

[0431] Input: Compressed photo data

[0432] Output: Data stored in the database, consistency check results

[0433] Step 7:

[0434] The server runs an image correction AI to convert the received monochrome photo into color, assigning the appropriate color to each pixel and reproducing natural color tones.

[0435] Input: Monochrome photo data

[0436] Output: Colorized photo data

[0437] How it works: An AI algorithm colors each image pixel by pixel, ensuring a natural overall color tone.

[0438] Step 8:

[0439] The server passes the colorized photo through a focus correction algorithm, which detects blurry areas and applies a sharpening filter.

[0440] Input: Colorized photo data

[0441] Output: Focused photo data

[0442] What it does: Image analysis identifies blurry areas and applies a sharpening filter.

[0443] Step 9:

[0444] The server starts the emotion engine and initializes the camera and microphone to analyze the user's voice and facial expressions.

[0445] Input: User voice and facial expression data

[0446] Output: Initial setup completed

[0447] What it does: Camera and microphone settings are automatically adjusted, and voice and facial expression data is captured in real time.

[0448] Step 10:

[0449] The device transmits the user's voice and facial expression data to the emotion engine in real time.

[0450] Input: User's voice and facial expression data

[0451] Output: Data sent to the emotion engine

[0452] Step 11:

[0453] The emotion engine analyzes the received data and determines the user's emotion.

[0454] Input: Voice and facial expression data

[0455] Output: Sentiment analysis result (e.g., happy, confused)

[0456] Specific operation: The AI ​​algorithm analyzes voice tone, facial features, etc. to determine emotions.

[0457] Step 12:

[0458] The server passes the judgment results from the emotion engine to the generation AI and works in conjunction with the generation AI.

[0459] Input: Sentiment analysis results

[0460] Output: Data passed to the generation AI

[0461] Specific operation: Emotional data is provided to the generative AI, which generates customizations based on the user's emotions.

[0462] Step 13:

[0463] The server uses the generated AI model to generate a dialogue interface with the user.

[0464] Input: Generative AI model held by the server

[0465] Output: Generates a conversational interface

[0466] Specific operation: Using natural language processing, it dynamically generates an easy-to-use dialogue interface for users.

[0467] Step 14:

[0468] The user inputs specific customization instructions through the dialogue interface.

[0469] Input: User customization instructions

[0470] Output: Instruction data sent to the server

[0471] Step 15:

[0472] The generative AI model analyzes the user's instructions and applies corrections to the initially processed photo.

[0473] Input: User customization instructions and initial processed photo data

[0474] Output: Photo data with corrections

[0475] What it does: Based on user instructions, the image editing algorithm applies specific modifications.

[0476] Step 16:

[0477] The emotion engine analyzes the user's emotions in real time and provides feedback to the generative AI.

[0478] Input: Real-time user emotion data

[0479] Output: Feedback provided to the generative AI

[0480] What it does: The updated emotion data is passed to the generative AI, providing reference information for further customization.

[0481] Step 17:

[0482] The server displays the results of the modifications in real time and asks the user for confirmation, allowing the user to make further adjustments while viewing the preview.

[0483] Input: Modified photo data and user emotion data

[0484] Output: Preview screen and user adjustment instructions

[0485] Step 18:

[0486] The server generates the final edited photo data and creates a permanent link.

[0487] Input: Final edited photo data

[0488] Output: Download link

[0489] Step 19:

[0490] The server stores the final photo data and provides the user with a download link via email or in-app notification.

[0491] Input: Last modified photo data and user contact information

[0492] Output: Provide a download link

[0493] Step 20:

[0494] The user clicks on the provided link and downloads the final photo data via a dedicated app or browser.

[0495] Input: Download link

[0496] Output: Downloaded photo data

[0497] Step 21:

[0498] Users can save the downloaded photos to their own devices and print them out or share them with family and friends as needed.

[0499] Input: Downloaded photo data

[0500] Output: Saved and shared photos

[0501] (Application example 2)

[0502] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0503] Conventional photo editing systems simply colorize and correct uniform photos without considering the user's emotions, making it difficult to achieve the optimal customization desired by the user. Furthermore, while the advertising industry requires the generation of advertising materials based on user emotions, current technology has difficulty providing this efficiently and effectively. Therefore, there is a need for a system that can analyze user emotions and customize photos based on them.

[0504] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0505] In this invention, the server includes means for taking or scanning old monochrome photographs, means for transmitting the taken or scanned monochrome photograph data to the server, means for performing image correction on the server to convert the monochrome photograph to color, means for correcting the focus of the colorized photograph to make it clearer, means for analyzing emotions from the user's voice and facial expressions, means for interacting with the user using a generative AI and customizing the photo based on the user's instructions and emotional analysis, and means for finally providing the customized photo to the user. This enables optimal photo customization and generation of advertising materials that take the user's emotions into consideration.

[0506] An "old black and white photograph" is a black and white photograph taken a long time ago, and does not contain color information.

[0507] "Means for photographing or scanning" refers to a device or system for digitizing photographs using a camera, scanner, etc.

[0508] "Means for transmitting to a server" refers to a function for transferring digital data to a server via the Internet or other communications network.

[0509] "Image enhancement" refers to digital processing to improve the quality of photographs, particularly color conversion and focus correction.

[0510] The "means for converting to color" is a technique for estimating the color of each pixel in a monochrome photograph and converting it into a color photograph with natural color tones.

[0511] "Focus correction and sharpening" is a technique that detects blurry areas in an image and applies a sharpening filter to make the photo clearer.

[0512] "Means for analyzing emotions from the user's voice and facial expressions" refers to software or algorithms that use a microphone or camera to collect data on the user's voice and facial expressions and analyze the emotions.

[0513] "Means for interacting with users using generative AI" refers to a system that utilizes artificial intelligence to interact with users and receive their instructions.

[0514] The "means for photo customization" is a system for photo editing, such as color adjustment and removal of unwanted background, based on user instructions and emotional analysis.

[0515] The "means for providing customized photos to users" is a mechanism for delivering the final processed photo data to users.

[0516] The present invention provides a system that colorizes old monochrome photos, corrects the focus, and makes them clearer, as well as analyzes the user's emotions and customizes the photos based on those emotions. This system is composed of a server and a terminal, and includes the following means:

[0517] First, the user prepares an old monochrome photograph and takes or scans it using a device (smartphone or tablet). At this time, the device displays a shooting guide that instructs the appropriate shooting angle and lighting conditions. The captured photograph is previewed, and if the user is satisfied with it, it is sent to the server.

[0518] The server temporarily stores the received photo data and verifies its integrity. It then runs an image correction algorithm to convert the monochrome photo to color, using image correction technologies such as OpenCV and TensorFlow. The colorized photo is then passed to a focus correction algorithm to sharpen any blurry areas.

[0519] The server then launches an emotion engine, using the camera and microphone to analyze the user's voice and facial expression data in real time. It performs emotion analysis using Google Cloud's Natural Language API. After determining the user's emotion, it uses a generative AI model (e.g., OpenAI's GPT-4) to generate a dialogue interface for the user. Through this interface, the user can provide specific customization instructions, such as "blur the background" or "change the color of the outfit to blue."

[0520] The emotion engine analyzes the user's emotions in real time and provides feedback to the generative AI, which then customizes the app based on the user's emotions. For example, if the user is using a wedding photo and the "joy" emotion is detected, the generative AI will suggest a warm-toned background.

[0521] As a concrete example, when colorizing and customizing wedding photos, if a user requests that the background be "warm toned," the following prompt sentence can be used for the generative AI model:

[0522] Example prompt sentence:

[0523] The user emotion is "joy." The provided photo is an old black-and-white wedding photo that has been colorized and focused. Create an ad using this photo as a base, suggesting a warm-toned background and an elegant tagline that will inspire users.

[0524] The server then applies the instructions and delivers the final, customized photo to the user. The photo data is then saved and a download link is provided to the user via email or in-app notification. The user can then download and use the final photo data via this link.

[0525] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0526] Step 1:

[0527] The user prepares an old black-and-white photograph and turns on the device (smartphone or tablet). The device displays a shooting guide, instructing on the appropriate shooting angle and lighting conditions. The user takes a photo and checks the preview. The user selects a photo they are satisfied with and presses the "upload" button in the dedicated app. At this point, the input is the digital data of the black-and-white photograph, and the output is photo data ready to be sent to the server.

[0528] Step 2:

[0529] The device compresses the selected photo data and sends it to the server. The input is the photo data stored on the device, and the output is the compressed photo data sent to the server. The server verifies the integrity of the received photo data and checks for omissions or errors.

[0530] Step 3:

[0531] The server runs image correction AI to convert received monochrome photos to color. It uses image correction technologies such as OpenCV and TensorFlow to estimate the color of each pixel and generate natural color tones. The input is monochrome photo data, and the output is colorized photo data.

[0532] Step 4:

[0533] The server passes the colorized photo data to a focus correction algorithm, which detects blurry areas in the image and applies a sharpening filter. The input is the colorized photo data, and the output is the sharpened photo data.

[0534] Step 5:

[0535] The server activates the emotion engine and uses the device's camera and microphone to analyze the user's voice and facial expressions. The device sends the user's voice and facial expression data to the emotion engine in real time. The input is the user's voice and facial expression data, and the output is the analyzed emotion data.

[0536] Step 6:

[0537] The server receives the judgment results from the emotion engine and connects with a generative AI model (e.g., OpenAI's GPT-4). The generative AI model generates a dialogue interface based on the user's emotions and instructions, and makes specific customization suggestions to the user. The input is emotion data and user instructions, and the output is customization suggestions.

[0538] Step 7:

[0539] The user provides customization instructions through a dialogue interface. The generative AI model analyzes the instructions and applies modifications to the initially processed photo. The input is the user's customization instructions, and the output is customized photo data.

[0540] Step 8:

[0541] The emotion engine analyzes the user's emotions in real time and provides feedback to the generative AI. The generative AI model then suggests additional modifications based on the emotions, resulting in optimal customization. The input is emotion data and user feedback, and the output is further modified photo data.

[0542] Step 9:

[0543] The server generates the final edited photo data and creates a permanent link. The server saves the final photo data and provides the user with a download link via email or in-app notification. The input is the final customized photo data, and the output is the download link.

[0544] Step 10:

[0545] The user clicks the provided link and downloads the final photo data through a dedicated app or browser. The user saves the downloaded photos to their device and prints them out or shares them with family and friends as needed. The input is the download link and the output is the downloaded photo data.

[0546] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0547] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0548] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0549] [Second embodiment]

[0550] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0551] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0552] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0553] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0554] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0555] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0556] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0557] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0558] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0559] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0560] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0561] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0562] The following describes in detail an embodiment of the present invention. The present invention is a system for converting old black-and-white photographs to color and sharpening out-of-focus photographs. This system is implemented through a dedicated app that is easy for users to operate, and includes the following processing flow:

[0563] Taking and uploading photos

[0564] 1. The user prepares an old black-and-white photo and launches the dedicated app.

[0565] 2. The device (smartphone or tablet) starts the camera and the user takes a photo. A shooting guide is displayed, instructing the user on the appropriate shooting angle and lighting conditions.

[0566] 3. The user can preview the photo they have taken and retake it if necessary.

[0567] 4. The user selects the photo they are satisfied with and presses the "Upload" button in the dedicated app.

[0568] 5. The device compresses the selected photo data and sends it to the server.

[0569] AI-based initial processing

[0570] 1. The server receives the photo data and runs the image correction AI. First, it converts the monochrome photo to color. This process involves an algorithm that colors each pixel to create natural-looking tones.

[0571] 2. A focus correction algorithm is then used to detect blurry areas in the colorized photo and a sharpening filter is applied, resulting in a clearer image.

[0572] Customization Process

[0573] 1. The server uses the generative AI model to generate a dialogue interface with the user, through which the user can provide specific customization instructions.

[0574] 2. Users can interact with the AI ​​to adjust the colors to their liking or remove unnecessary background elements. For example, they can communicate specific requests such as "I want the background blurred" or "I want the costume color changed to blue."

[0575] 3. The generative AI model analyzes the user's instructions and applies real-time corrections to the initially processed photo.

[0576] 4. The server displays a preview of the edited result to the user and asks for confirmation. The user can make further adjustments while viewing the preview.

[0577] Final data storage and distribution

[0578] 1. The server generates the final edited photo data and creates a permanent link.

[0579] 2. The server saves the final photo data and provides the user with a download link via email or in-app notification.

[0580] 3. The user clicks on the provided link and downloads the final photo data via a dedicated app or browser.

[0581] 4. Users can save the downloaded photos to their own devices and print them out or share them with family and friends as needed.

[0582] Specific examples

[0583] Example 1: Colorizing a black and white wedding photo

[0584] 1. The user launches the app and takes an old black-and-white wedding photo with their camera.

[0585] 2. The device sends the captured image data to the server.

[0586] 3. The server colorizes and sharpens the received photo.

[0587] 4. The server interacts with the user through the generated AI model and receives instructions on adjusting the background.

[0588] 5. The server generates the final retouched photo and provides the user with a download link.

[0589] 6. User downloads and saves the final photo via the link.

[0590] Example 2: Sharpening an out-of-focus photo of grandparents

[0591] 1. The user digitizes old photos using a scanner and uploads them using a dedicated app.

[0592] 2. The device sends the scanned data to the server.

[0593] 3. The server receives the data and performs focus correction and colorization processing.

[0594] 4. The server interacts with the generated AI model and modifies facial details based on the user's wishes.

[0595] 5. The server generates the final photo after user confirmation and provides a download link.

[0596] 6. Users download, save and share.

[0597] By implementing this invention, old black-and-white photographs can be easily colorized and out-of-focus photographs can be sharpened, allowing users to vividly relive past memories. This makes the system extremely useful, especially for people in their 50s and older.

[0598] The processing flow will be explained below.

[0599] Step 1:

[0600] The user prepares an old black-and-white photo and launches the dedicated app.

[0601] Step 2:

[0602] The device (smartphone or tablet) starts the camera and the user takes a photo. A shooting guide is displayed, instructing the appropriate shooting angle and lighting conditions.

[0603] Step 3:

[0604] The user can preview the photo they have taken and take another photo if necessary.

[0605] Step 4:

[0606] The user selects the photo they are satisfied with and presses the "Upload" button in the dedicated app.

[0607] Step 5:

[0608] The terminal compresses the selected photo data and transmits it to the server.

[0609] Step 6:

[0610] The server receives the photo data and temporarily stores it in an input database, where it checks the data for consistency and for missing or error information.

[0611] Step 7:

[0612] The server runs an image correction AI to convert the received monochrome photo into color, which then colors each pixel to produce a natural color tone.

[0613] Step 8:

[0614] The server passes the colorized photo to a focus correction algorithm, which detects blurry areas and applies a sharpening filter.

[0615] Step 9:

[0616] The server sends the color photo data after initial processing to the generative AI model.

[0617] Step 10:

[0618] The server generates an interactive interface with the user through the generated AI model.

[0619] Step 11:

[0620] The user can use the dialogue interface to make color adjustments or remove unnecessary backgrounds, for example, by inputting instructions such as "I want the background to be blurred" or "I want the color of the costumes to be changed to blue."

[0621] Step 12:

[0622] The generative AI model analyzes the user's instructions and applies corrections to the initially processed photo.

[0623] Step 13:

[0624] The server displays the correction results in real time and asks the user for confirmation, allowing the user to give further instructions while viewing the preview and adjust the final result.

[0625] Step 14:

[0626] The server generates the final edited photo data and creates a permanent link.

[0627] Step 15:

[0628] The server stores the final photo data and provides the user with a download link via email or in-app notification.

[0629] Step 16:

[0630] The user clicks on the provided link and downloads the final photo data via a dedicated app or browser.

[0631] Step 17:

[0632] Users can save the downloaded photos to their own devices and print them out or share them with family and friends as needed.

[0633] Example 1

[0634] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0635] Conventional photo correction systems often require manual work, whether it's converting old black-and-white photos to color or correcting out-of-focus photos to make them clearer, making the process extremely time-consuming. Furthermore, specific corrections and customizations require users to have advanced skills, making them difficult for average users without specialized knowledge. For this reason, there is a demand for a system that allows users to easily perform high-quality photo corrections.

[0636] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0637] In this invention, the server includes a device that takes or scans old black-and-white photographs, a device that transmits the taken or scanned black-and-white photograph data to the server, a program that performs image correction on the server and converts the black-and-white photographs to color, a program that corrects the focus of the colorized photographs to make them clearer, a program that interacts with the user using a generative AI model and customizes the photographs based on the user's instructions, and a device that finally provides the customized photographs to the user. This allows users to easily perform high-quality photo correction without specialized knowledge.

[0638] An "old black and white photograph" is a black and white (monochrome) photograph taken in the past, and refers to an image that does not contain color information.

[0639] A "photographing or scanning device" is a device for converting a physical photograph into a digital image, and includes hardware devices such as a camera or scanner.

[0640] "Transmitting device" refers to a hardware or software device for transferring photographed or scanned digital image data to a server via a network.

[0641] "Server" refers to a remote computer or cloud-based computing system for receiving and processing digital image data.

[0642] "Image correction program" refers to a software application that performs operations such as adjusting color tone and correcting exposure on digital images.

[0643] A "color conversion program" refers to a software algorithm that adds color information to each pixel of a monochrome image to convert it into a natural color image.

[0644] A "focus and sharpening program" refers to software that detects blurry areas in an image and applies a sharpening filter to improve the image.

[0645] "Generative AI model" refers to an artificial intelligence model that interacts with users, analyzes their instructions based on natural language prompts, and is responsible for making corrections and improvements to images.

[0646] "Customization program" refers to software that modifies images based on a user's specific needs.

[0647] The "providing device" refers to a device for distributing the final edited image data to the user, and includes email, cloud storage services, etc.

[0648] The system of this invention is mainly designed to convert old black and white photos into color and sharpen out-of-focus photos, allowing users to vividly relive their past memories. The whole system consists of three main components: the user, the terminal, and the server.

[0649] First, the user prepares an old black-and-white photograph and launches the dedicated application. This application runs on devices such as smartphones and tablets, and activates the camera to take a photo. Shooting guides are displayed, allowing the user to take a photo with the optimal lighting and angle.

[0650] The device then displays a preview of the captured photo, allowing the user to review and retake the photo until satisfied. Once a satisfactory photo is selected, the device compresses the photo data and sends it to the server.

[0651] The server receives the photo data and performs image correction. The image correction process first converts the monochrome photo to color. Specifically, it uses Python's OpenCV library to apply an algorithm that estimates the appropriate color for each pixel and generates natural-looking tones. Next, a focus correction algorithm using TensorFlow is applied, detecting blurred areas and applying a sharpening filter to make the entire photo clearer.

[0652] In addition, a customization process is provided in which users interact with a generative AI model. This generative AI model uses advanced natural language processing techniques, such as GPT-4, to modify photos based on prompts entered by the user. Users can provide specific customization instructions through the dialogue interface, such as "blur the background" or "change the color of the outfit to blue." The generative AI model analyzes these instructions and applies them to the photo in real time.

[0653] The final customized photo is generated as high-resolution image data by the server. The server stores this data in cloud storage and generates a permanent link that is provided to the user. The user receives a download link via email or in-app notification, and can download the final photo data through a dedicated application or browser and save it to their device. This allows the user to print out the final photo or share it with family and friends.

[0654] As a concrete example, we will show the process of colorizing a black-and-white wedding photo. The user launches a dedicated app and takes a photo of an old black-and-white wedding photo with their camera. The device sends the captured data to a server, which then colorizes and sharpens the received photo. The background is then adjusted based on the user's instructions via a generative AI model, and the user is provided with a download link for the final edited photo.

[0655] By implementing this invention, users can easily colorize old monochrome photographs and sharpen out-of-focus photographs without any specialized knowledge. This provides a very useful system for vividly reviving past memories, especially for those in their 50s and older.

[0656] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0657] Step 1:

[0658] The user prepares an old black-and-white photo and launches the dedicated app.

[0659] Input: Physical black and white photo

[0660] Output: Launching the dedicated app

[0661] Specific operation: The user prepares a physical black-and-white photo and launches a dedicated app on their smartphone or tablet.

[0662] Step 2:

[0663] The device will activate the camera and take a photo.

[0664] Input: User-initiated camera activation and shooting instructions

[0665] Output: Digitized black and white photo capture

[0666] Specific operation: The device's camera is activated, a shooting guide is displayed, and the user presses the shooting button to digitize a monochrome photo.

[0667] Step 3:

[0668] The device displays a preview of the photo for the user to review and retake.

[0669] Input: Digitized black and white photograph

[0670] Output: Show preview image

[0671] Specific operation: The device displays a preview of the photo taken and displays the "Confirm" and "Retake" buttons. The user checks the preview and presses the "Retake" button if they need to take another photo.

[0672] Step 4:

[0673] The user selects a photo they are satisfied with and presses the upload button.

[0674] Input: A preview image selected by the user.

[0675] Output: Ready to upload status

[0676] Specific operation: The user presses the "Confirm" button, which takes them to the upload screen of the dedicated app. The user presses the "Upload" button.

[0677] Step 5:

[0678] The terminal compresses the selected photo data and transmits it to the server.

[0679] Input: A digitized black and white photograph selected by the user

[0680] Output: Compressed photo data sent to the server

[0681] Specific operation: The device compresses the image data and sends it to a server via the Internet.

[0682] Step 6:

[0683] The server receives the photo data and performs image correction.

[0684] Input: Compressed photo data

[0685] Output: Colorized and focused image

[0686] What it does: The server receives the photo data and colorizes the monochrome photo using Python's OpenCV library, then uses TensorFlow to apply a focus correction algorithm to make the photo clearer.

[0687] Step 7:

[0688] The server uses the generative AI model to generate a dialogue interface with the user.

[0689] Input: Colorized and focus-corrected image data

[0690] Output: Generate a conversational interface

[0691] Specific operation: The server uses a generative AI model (e.g., GPT-4) to generate a dialogue interface and provides a screen for the user to input customization instructions.

[0692] Step 8:

[0693] The user gives specific customization instructions.

[0694] Input: Prompt text given by the user (e.g., "I want the background to be blurred" or "I want the color of the costume to be blue")

[0695] Output: Instruction analysis results by the generative AI model

[0696] Specific operation: The user inputs a prompt sentence through the dialogue interface, and the AI ​​receives and analyzes it.

[0697] Step 9:

[0698] A generative AI model modifies photos based on user instructions.

[0699] Input: Prompt statement analysis result

[0700] Output: Corrected image data

[0701] How it works: The generative AI model modifies specific parts of the image based on user instructions and makes corresponding adjustments in real time.

[0702] Step 10:

[0703] The server displays a preview of the correction results to the user.

[0704] Input: Corrected image data

[0705] Output: Preview screen display

[0706] What happens: The server previews the corrected image to the user, allowing the user to make further adjustments.

[0707] Step 11:

[0708] The server generates the final edited photo data and creates a fixed link.

[0709] Input: Final corrected image data

[0710] Output: Cloud storage of image data and generation of a permanent link

[0711] Specific operation: The server generates a high-resolution image after correction is completed, saves it to cloud storage, and then creates a permanent link.

[0712] Step 12:

[0713] The server notifies the link and provides it to the user.

[0714] Input:permalink

[0715] Output: User notification (email or in-app notification)

[0716] Specific operation: The server notifies the user of the fixed link and provides a download link.

[0717] Step 13:

[0718] The user clicks on the provided link to download the final photo data.

[0719] Input:permalink

[0720] Output: Final downloaded photo data

[0721] Specific operation: The user clicks on the link and downloads the final photo data via a dedicated app or web browser.

[0722] Step 14:

[0723] Save the photos you download to your device and share them as needed.

[0724] Input: Downloaded photo data

[0725] Output: Photos stored on the device and shared photos

[0726] What it does: Users can save the downloaded photos to their devices, share them with family and friends, and even print them out.

[0727] (Application example 1)

[0728] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0729] In recent years, advances in digital technology have led to an increasing number of users wanting to digitize and preserve old photographs. However, converting old black-and-white photographs to color and sharpening blurry photographs is a time-consuming process that requires specialized knowledge. Furthermore, there are limited means to easily provide such services, especially in brick-and-mortar stores. This makes it difficult for customers to restore photos at their own convenience, leading to lower service satisfaction. Therefore, there is a need for a system that can easily colorize and sharpen old photographs in-store.

[0730] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0731] In this invention, the server includes means for taking or scanning old monochrome photographs, means for transmitting the taken or scanned monochrome photograph data to the server, means for performing image correction on the server to convert the monochrome photograph to color, means for correcting the focus of the colorized photograph to make it clearer, means for interacting with the user using a generative AI and customizing the photograph based on the user's instructions, means for providing the user with the final customized photograph, and means for the user to use the application that performs the photo customization process in a physical store environment. This allows customers to easily colorize and sharpen old photographs in a physical store.

[0732] An "old black-and-white photograph" is a photograph taken in the past that lacks color and is expressed in black and white rather than color.

[0733] "Means of photographing or scanning" refers to the use of a device such as a camera or scanner to obtain a photograph as digital data.

[0734] A "transmitting means" is a communication means for sending digital data over the Internet or a local network to a remote server.

[0735] A "server" is a computer system that receives and processes digital data.

[0736] "Image correction means" refers to software or algorithms for adjusting contrast and correcting color of digital images.

[0737] "Methods for converting to color" are techniques for adding color to monochrome photographs to reproduce them as natural color photographs.

[0738] "Focus correction and sharpening" is a method for sharpening blurry photos and improving the clarity of details.

[0739] "Generative AI" is a system that uses an artificial intelligence model to interact with users in natural language and process images based on their explanations.

[0740] "Means for interacting with users and customizing" refers to technology that uses AI to understand users' requests and modify or process specific parts of images.

[0741] A "means for providing a customized photograph to a user" is a method by which the final, enhanced digital image can be downloaded or viewed by the user.

[0742] "Methods available to users in a physical store environment" refers to the use of applications and digital devices by customers to receive services within a physical store.

[0743] The following describes in detail the mode for carrying out the invention. This system has various functions for converting old black-and-white photos to color and sharpening out-of-focus photos. It is intended for use in brick-and-mortar stores, and customers can easily use these functions using a dedicated application.

[0744] Hardware and software used

[0745] The system uses the following hardware and software:

[0746] Hardware:

[0747] Devices (smartphones, tablets, cameras)

[0748] Server (high performance processor, internet connection)

[0749] software:

[0750] OpenCV (image capture, image processing)

[0751] PIL (Python Imaging Library) (Image Processing)

[0752] requests (HTTP communication library)

[0753] Generative AI model (user interaction, image customization)

[0754] Processing flow explanation

[0755] Taking and uploading photos

[0756] The user launches a dedicated app and prepares an old black-and-white photo. The device then activates the camera, and the user takes a photo. A shooting guide is displayed, instructing on the appropriate shooting angle and lighting conditions. The user can check a preview of the photo they have taken and take another photo if necessary. When the user selects a photo they are satisfied with and presses the upload button, the device compresses the photo data and sends it to the server.

[0757] AI-based initial processing

[0758] The server receives the photo data and runs an image correction AI, first converting the black and white photo to color, then using a focus correction algorithm to detect blurry areas in the colorized photo and applying a sharpening filter.

[0759] Customization Process

[0760] The server uses the generative AI model to generate a dialogue interface for the user. The user can use this interface to give specific customization instructions. For example, they can express specific requests such as "I want the background blurred" or "I want the costume color changed to blue." The generative AI model analyzes the user's instructions and applies real-time corrections to the initially processed photo. The server then displays a preview of the corrections to the user and receives confirmation and additional instructions.

[0761] Final data storage and distribution

[0762] The server generates the final edited photo data and creates a permanent link. The server saves the final photo data and provides the user with a download link via email or in-app notification. The user clicks the link and downloads the final photo data via a dedicated app or browser. The user saves the downloaded photo to their device and can print it out or share it with family and friends as needed.

[0763] Examples of concrete examples and prompts

[0764] As a concrete example, customers can take old family photos with the app in a physical store where their families gather, and have them colorized and sharpened on the spot. This service allows customers to vividly relive their past memories and share them as new memories.

[0765] Example prompt sentence:

[0766] 1. Take a photo or upload an existing one.

[0767] 2. Sending photos to server... Please wait.

[0768] 3. Processing the photo… We will correct the focus and add color to the original photo.

[0769] 4. Processing completed! Please check.

[0770] 5. Do you want to customize further? For example, "I want the background to be blurred" or "I want the costume color to be changed to blue."

[0771] This system allows customers to easily colorize and sharpen old photos in-store.

[0772] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0773] Step 1:

[0774] Photographing and scanning

[0775] The user prepares an old black-and-white photo and launches a dedicated app. The device launches the camera and the user takes a photo. A shooting guide is displayed, instructing on the appropriate shooting angle and lighting conditions. The input is the old black-and-white photo, and the output is a digital copy of the photo. A preview is provided to confirm that the appropriate shooting conditions are met, allowing the user to decide whether or not to retake the photo.

[0776] Step 2:

[0777] Upload

[0778] When the user selects a photo they are satisfied with and presses the upload button, the device compresses the photo data and sends it to the server. The input is a digital photo, and the output is compressed photo data. This process improves data transfer speed and reduces server load.

[0779] Step 3:

[0780] Initial Data Processing

[0781] The server receives the photo data and first runs the image correction AI. The input is compressed photo data, and the output is colorized photo data. Specifically, it applies an algorithm that adds color to each pixel of the monochrome photo and generates natural-looking tones.

[0782] Step 4:

[0783] Focus Correction

[0784] The server then uses a focus correction algorithm to detect blurred areas in the colorized photo and applies a sharpening filter. The input is the colorized photo data, and the output is the sharpened photo data. Specifically, the server runs a filter to improve the contrast and detail in the blurred areas.

[0785] Step 5:

[0786] Customization with generative AI

[0787] The server uses the generative AI model to generate a dialogue interface for the user. The user can then provide specific customization instructions through this interface. The input is the user's text-based instructions, and the output is customized photo data. Specific operations include analyzing requests such as "blur the background" or "change the color of the costume to blue," and reflecting the results in image processing.

[0788] Step 6:

[0789] Preview and fix

[0790] The server displays a preview of the correction results to the user and receives confirmation and additional instructions from the user. The input is customized photo data, and the output is the final corrected photo data. Specifically, it provides an interface that allows the user to make further adjustments while viewing the preview.

[0791] Step 7:

[0792] Final data storage and distribution

[0793] The server generates the final corrected photo data and creates a permanent link. The server saves the final photo data and provides the user with a download link via email or in-app notification. The input is the final corrected photo data and the output is a permanent link. Specifically, this involves saving the photo in a database and generating a link to provide it to the user.

[0794] Through this process, customers can easily colorize and sharpen old photos in-store.

[0795] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0796] The following describes in detail the embodiments of the present invention. The present invention combines a system for colorizing old black-and-white photos and sharpening out-of-focus photos with an emotion engine that recognizes the user's emotions. This system is implemented through a dedicated app that is easy for users to operate, and includes the following processing flow:

[0797] Taking and uploading photos

[0798] 1. The user prepares an old black-and-white photo and launches the dedicated app.

[0799] 2. The device (smartphone or tablet) starts the camera and the user takes a photo. A shooting guide is displayed, instructing the user on the appropriate shooting angle and lighting conditions.

[0800] 3. The user can preview the photo they have taken and retake it if necessary.

[0801] 4. The user selects the photo they are satisfied with and presses the "Upload" button in the dedicated app.

[0802] 5. The device compresses the selected photo data and sends it to the server.

[0803] AI-based initial processing

[0804] 1. The server receives the photo data and temporarily stores it in an input database, where it checks the data for consistency and omissions or errors.

[0805] 2. The server runs the image correction AI to convert the received monochrome photo to color. The image correction AI colors each pixel to produce natural-looking colors.

[0806] 3. The server passes the colorized photo to a focus correction algorithm, which detects blurry areas and applies a sharpening filter.

[0807] Emotion engine processing

[0808] 1. The server activates the emotion engine and uses the camera and microphone to analyze the user's voice and facial expressions.

[0809] 2. The device transmits the user's voice and facial expression data to the emotion engine in real time.

[0810] 3. The emotion engine analyzes the received data and determines the user's emotion. For example, if the user is smiling, it is determined to be "joy," and if they are frowning, it is determined to be "confusion."

[0811] 4. The server receives the judgment results from the emotion engine and works with the generation AI.

[0812] Customization Process

[0813] 1. The server uses the generative AI model to generate a dialogue interface with the user.

[0814] 2. The user enters specific customization instructions through the dialogue interface, such as "I want the background to be blurred" or "I want the color of the costume to be blue."

[0815] 3. The generative AI model analyzes the user's instructions and applies corrections to the initially processed photo.

[0816] 4. The emotion engine analyzes the user's emotions in real time and provides feedback to the generative AI, allowing the generative AI model to suggest additional modifications based on the emotions.

[0817] 5. The server displays the edited results in real time and asks the user for confirmation. The user can make further adjustments while viewing the preview.

[0818] Final data storage and distribution

[0819] 1. The server generates the final edited photo data and creates a permanent link.

[0820] 2. The server saves the final photo data and provides the user with a download link via email or in-app notification.

[0821] 3. The user clicks on the provided link and downloads the final photo data via a dedicated app or browser.

[0822] 4. Users can save the downloaded photos to their own devices and print them out or share them with family and friends as needed.

[0823] Specific examples

[0824] Example 1: Colorization of Black and White Wedding Photos and Emotion-Based Customization

[0825] 1. The user launches the app and takes an old black-and-white wedding photo with their camera.

[0826] 2. The device sends the captured image data to the server.

[0827] 3. The server colorizes and sharpens the received photo.

[0828] 4. The server interacts with the user through the generative AI model and emotion engine and receives instructions on adjusting the background.

[0829] 5. The emotion engine detects the user's happiness and suggests warmer background colors.

[0830] 6. The server generates the final retouched photo and provides the user with a download link.

[0831] 7. User downloads and saves the final photo via the link.

[0832] Example 2: Sharpening out-of-focus photos of grandparents and emotion-based customization

[0833] 1. The user digitizes old photos using a scanner and uploads them using a dedicated app.

[0834] 2. The device sends the scanned data to the server.

[0835] 3. The server receives the data and performs focus correction and colorization processing.

[0836] 4. The server interacts with the user through the generative AI model and emotion engine, and performs facial detail correction.

[0837] 5. The emotion engine analyzes the user's emotions, and if it detects "confusion," the generative AI will suggest more detailed adjustment methods.

[0838] 6. The server generates the final photo after user confirmation and provides a download link.

[0839] 7. Users can download, save, or share.

[0840] By implementing this invention, not only can old black-and-white photographs be easily colorized and out-of-focus photographs be made clearer, but the system can also be optimally customized based on the user's emotions, allowing users to relive their past memories in a richer way. This makes the system extremely useful, especially for those in their 50s and older.

[0841] The processing flow will be explained below.

[0842] Step 1:

[0843] The user prepares an old black-and-white photo and launches the dedicated app.

[0844] Step 2:

[0845] The device (smartphone or tablet) starts the camera and the user takes a photo. A shooting guide is displayed, instructing the appropriate shooting angle and lighting conditions.

[0846] Step 3:

[0847] The user can preview the photo they have taken and take another photo if necessary.

[0848] Step 4:

[0849] The user selects the photo they are satisfied with and presses the "Upload" button in the dedicated app.

[0850] Step 5:

[0851] The terminal compresses the selected photo data and transmits it to the server.

[0852] Step 6:

[0853] The server receives the photo data and temporarily stores it in an input database, where it checks the data for consistency and for missing or error information.

[0854] Step 7:

[0855] The server runs an image correction AI to convert the received monochrome photo into color, which then colors each pixel to produce a natural color tone.

[0856] Step 8:

[0857] The server passes the colorized photo to a focus correction algorithm, which detects blurry areas and applies a sharpening filter.

[0858] Step 9:

[0859] The server runs the emotion engine and uses the camera and microphone to analyze the user's voice and facial expressions.

[0860] Step 10:

[0861] The device transmits the user's voice and facial expression data to the emotion engine in real time.

[0862] Step 11:

[0863] The emotion engine analyzes the received data and determines the user's emotion. For example, if the user is smiling, it is determined to be "joy," and if they are frowning, it is determined to be "confusion."

[0864] Step 12:

[0865] The server receives the judgment results from the emotion engine and works in conjunction with the generation AI.

[0866] Step 13:

[0867] The server uses the generative AI model to generate a dialogue interface with the user.

[0868] Step 14:

[0869] The user enters specific customization instructions through a dialogue interface, such as "I want the background to be blurred" or "I want the color of the costume to be changed to blue."

[0870] Step 15:

[0871] The generative AI model analyzes the user's instructions and applies corrections to the initially processed photo.

[0872] Step 16:

[0873] The emotion engine analyzes the user's emotions in real time and provides feedback to the generative AI.

[0874] Step 17:

[0875] The generative AI model suggests additional modifications based on the user's sentiment.

[0876] Step 18:

[0877] The server displays the correction results in real time and asks the user for confirmation, allowing the user to make further adjustments while viewing the preview.

[0878] Step 19:

[0879] The server generates the final edited photo data and creates a permanent link.

[0880] Step 20:

[0881] The server stores the final photo data and provides the user with a download link via email or in-app notification.

[0882] Step 21:

[0883] The user clicks on the provided link and downloads the final photo data via a dedicated app or browser.

[0884] Step 22:

[0885] Users can save the downloaded photos to their own devices and print them out or share them with family and friends as needed.

[0886] Example 2

[0887] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0888] Conventional photo restoration systems provide methods for colorizing old monochrome photographs and correcting focus, but do not offer customization that takes into account the user's emotions. As a result, it takes a lot of time and effort to achieve results that satisfy the user. The purpose of this invention is to solve these problems and provide a system that is easy for users to operate and that allows customization based on emotions.

[0889] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0890] In this invention, the server includes means for taking or scanning old monochrome photographs, means for transmitting the taken or scanned monochrome photograph data to an information processing device, means for performing image correction on the information processing device to convert the monochrome photograph to color, means for correcting the focus of the colorized photograph to make it clearer, means for analyzing the user's voice and facial expression in real time to determine their emotions, means for interacting with the user using a generative AI and customizing the photo based on the user's instructions and the determined emotions, and means for finally providing the customized photo to the user. This enables photo restoration and customization that satisfies the user to be performed quickly and easily.

[0891] "Old monochrome photographs" are black and white photographic data that were taken in the past and do not contain color information.

[0892] "Photographing or scanning means" refers to methods and devices that use cameras or scanners to convert physical photographs or images into digital data.

[0893] "Information processing device" refers to a computer system such as a server or cloud computing system for processing captured or scanned image data.

[0894] "Method of image correction and converting black-and-white photos to color" refers to an image processing technology that assigns an appropriate color to each pixel of a black-and-white photo to reproduce natural color tones.

[0895] "Method for correcting focus and sharpening colorized photographs" refers to a technique that detects blurry areas of an image and applies a sharpening filter to them to increase the overall clarity of the image.

[0896] "Means for analyzing a user's voice and facial expressions in real time to determine emotions" refers to a system that uses a microphone or camera to collect a user's voice and facial expressions, and analyzes that data to identify the user's emotional state.

[0897] "Generative AI" refers to an artificial intelligence system that interacts with users based on input data and instructions and automatically performs the necessary customizations.

[0898] "Means for interacting with a user and customizing a photo based on the user's instructions and determined emotions" refers to a method and apparatus that receives instructions from a user via a user interface and further processes or modifies a photo taking into account the user's emotional state.

[0899] "Means for providing the final customized photo to the user" means a system that provides the final processed photo data to the user via a download link or in-app notification.

[0900] This system aims to restore a clear image by colorizing old monochrome photographs and adjusting the focus. It also analyzes the user's voice and facial expressions to determine their emotions, and customizes the photo restoration process based on those emotions to ensure the user is satisfied with the results. This system is implemented using the following hardware and software:

[0901] Hardware configuration:

[0902] Device: A device such as a smartphone, tablet, or computer that contains a camera and microphone and collects user input and emotional data.

[0903] Server: A server with high-performance processing power, where image processing, database management, and AI model execution take place.

[0904] Software configuration:

[0905] Image Enhancement Software: This includes AI algorithms that colorize old black-and-white photos and correct focus. Specific examples of the software used include OpenCV and TensorFlow.

[0906] Emotion engine: Software that analyzes the user's voice and facial expressions to determine their emotions. Specific software used includes "DeepFace" and "Google Cloud Speech-to-Text."

[0907] Generative AI models: Personalize photos based on user input and emotions, using natural language processing and generative models like GPT-3 and DALL·E.

[0908] Examples:

[0909] 1. Colorization of monochrome wedding photos and emotion-based customization

[0910] The user launches the dedicated app and takes an old black-and-white wedding photo with their camera.

[0911] The device sends the captured image data to the server.

[0912] The server then colorizes and sharpens the received photographs.

[0913] The server interacts with the user through the generative AI model and emotion engine, and receives instructions for adjusting the background.

[0914] The emotion engine detects the user's happiness and suggests warmer background colors.

[0915] The server generates the final retouched photo and provides the user with a download link.

[0916] The user downloads and saves the final photo via a link.

[0917] 2. Sharpening out-of-focus photos of grandparents and emotion-based customization

[0918] Users digitize old photos using a scanner and upload them using a dedicated app.

[0919] The device sends the scanned data to the server.

[0920] The server receives the data and performs focus correction and colorization processing.

[0921] The server interacts with the user through the generative AI model and emotion engine, and performs facial detail correction.

[0922] The emotion engine analyzes the user's emotions, and if it detects "confusion," the generative AI will suggest more detailed adjustment methods.

[0923] The server generates the final photo after user confirmation and provides a download link.

[0924] Users can download, save and share.

[0925] Example prompt sentence:

[0926] 1. "Blur the background in this photo to make the person stand out."

[0927] 2. "Please change the color of the outfit in this photo to blue to balance the overall look."

[0928] 3. "I'd like to see a bit more sharpness in this photo to bring out the details."

[0929] This system allows users to easily colorize old black-and-white photos, improve their clarity, and even customize them individually, allowing users to more vividly relive their rich memories.

[0930] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0931] Step 1:

[0932] The user prepares an old black-and-white photo and launches the dedicated app.

[0933] Input: Old black and white photo

[0934] Output: Ready to be photographed or scanned

[0935] Step 2:

[0936] When the user presses the "Take Photo" button on the dedicated app, the device's camera will launch and a photo guide will be displayed.

[0937] Input: User action (pressing the "Take Photo" button)

[0938] Output: Starts the camera and displays the shooting guide

[0939] What it does: The shooting guide displays messages instructing you on the appropriate shooting angles and lighting conditions.

[0940] Step 3:

[0941] The user takes a photo using the camera and then sees a preview screen after taking the photo, which allows them to choose whether to retake the photo or use the one they just took.

[0942] Input: Photo data taken

[0943] Output: Preview screen and user choice (retake or use)

[0944] Step 4:

[0945] The user selects a photo they are satisfied with and presses the "Upload" button in the dedicated app.

[0946] Input: User operation (pressing the "Upload" button)

[0947] Output: Photo ready to upload

[0948] Step 5:

[0949] The device compresses the selected photo data and sends it to the server. The device uses a compression algorithm to minimize the data size and reduce transmission time.

[0950] Input: Photo data taken

[0951] Output: Compressed photo data, sent to server

[0952] Step 6:

[0953] The server receives the photo data and temporarily stores it in an input database, where it checks the data for consistency and omissions or errors.

[0954] Input: Compressed photo data

[0955] Output: Data stored in the database, consistency check results

[0956] Step 7:

[0957] The server runs an image correction AI to convert the received monochrome photo into color, assigning the appropriate color to each pixel and reproducing natural color tones.

[0958] Input: Monochrome photo data

[0959] Output: Colorized photo data

[0960] How it works: An AI algorithm colors each image pixel by pixel, ensuring a natural overall color tone.

[0961] Step 8:

[0962] The server passes the colorized photo through a focus correction algorithm, which detects blurry areas and applies a sharpening filter.

[0963] Input: Colorized photo data

[0964] Output: Focused photo data

[0965] What it does: Image analysis identifies blurry areas and applies a sharpening filter.

[0966] Step 9:

[0967] The server starts the emotion engine and initializes the camera and microphone to analyze the user's voice and facial expressions.

[0968] Input: User voice and facial expression data

[0969] Output: Initial setup completed

[0970] What it does: Camera and microphone settings are automatically adjusted, and voice and facial expression data is captured in real time.

[0971] Step 10:

[0972] The device transmits the user's voice and facial expression data to the emotion engine in real time.

[0973] Input: User's voice and facial expression data

[0974] Output: Data sent to the emotion engine

[0975] Step 11:

[0976] The emotion engine analyzes the received data and determines the user's emotion.

[0977] Input: Voice and facial expression data

[0978] Output: Sentiment analysis result (e.g., happy, confused)

[0979] Specific operation: The AI ​​algorithm analyzes voice tone, facial features, etc. to determine emotions.

[0980] Step 12:

[0981] The server passes the judgment results from the emotion engine to the generation AI and works in conjunction with the generation AI.

[0982] Input: Sentiment analysis results

[0983] Output: Data passed to the generation AI

[0984] Specific operation: Emotional data is provided to the generative AI, which generates customizations based on the user's emotions.

[0985] Step 13:

[0986] The server uses the generated AI model to generate a dialogue interface with the user.

[0987] Input: Generative AI model held by the server

[0988] Output: Generates a conversational interface

[0989] Specific operation: Using natural language processing, it dynamically generates an easy-to-use dialogue interface for users.

[0990] Step 14:

[0991] The user inputs specific customization instructions through the dialogue interface.

[0992] Input: User customization instructions

[0993] Output: Instruction data sent to the server

[0994] Step 15:

[0995] The generative AI model analyzes the user's instructions and applies corrections to the initially processed photo.

[0996] Input: User customization instructions and initial processed photo data

[0997] Output: Photo data with corrections

[0998] What it does: Based on user instructions, the image editing algorithm applies specific modifications.

[0999] Step 16:

[1000] The emotion engine analyzes the user's emotions in real time and provides feedback to the generative AI.

[1001] Input: Real-time user emotion data

[1002] Output: Feedback provided to the generative AI

[1003] What it does: The updated emotion data is passed to the generative AI, providing reference information for further customization.

[1004] Step 17:

[1005] The server displays the results of the modifications in real time and asks the user for confirmation, allowing the user to make further adjustments while viewing the preview.

[1006] Input: Modified photo data and user emotion data

[1007] Output: Preview screen and user adjustment instructions

[1008] Step 18:

[1009] The server generates the final edited photo data and creates a permanent link.

[1010] Input: Final edited photo data

[1011] Output: Download link

[1012] Step 19:

[1013] The server stores the final photo data and provides the user with a download link via email or in-app notification.

[1014] Input: Last modified photo data and user contact information

[1015] Output: Provide a download link

[1016] Step 20:

[1017] The user clicks on the provided link and downloads the final photo data via a dedicated app or browser.

[1018] Input: Download link

[1019] Output: Downloaded photo data

[1020] Step 21:

[1021] Users can save the downloaded photos to their own devices and print them out or share them with family and friends as needed.

[1022] Input: Downloaded photo data

[1023] Output: Saved and shared photos

[1024] (Application example 2)

[1025] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1026] Conventional photo editing systems simply colorize and correct uniform photos without considering the user's emotions, making it difficult to achieve the optimal customization desired by the user. Furthermore, while the advertising industry requires the generation of advertising materials based on user emotions, current technology has difficulty providing this efficiently and effectively. Therefore, there is a need for a system that can analyze user emotions and customize photos based on them.

[1027] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1028] In this invention, the server includes means for taking or scanning old monochrome photographs, means for transmitting the taken or scanned monochrome photograph data to the server, means for performing image correction on the server to convert the monochrome photograph to color, means for correcting the focus of the colorized photograph to make it clearer, means for analyzing emotions from the user's voice and facial expressions, means for interacting with the user using a generative AI and customizing the photo based on the user's instructions and emotional analysis, and means for finally providing the customized photo to the user. This enables optimal photo customization and generation of advertising materials that take the user's emotions into consideration.

[1029] An "old black and white photograph" is a black and white photograph taken a long time ago, and does not contain color information.

[1030] "Means for photographing or scanning" refers to a device or system for digitizing photographs using a camera, scanner, etc.

[1031] "Means for transmitting to a server" refers to a function for transferring digital data to a server via the Internet or other communications network.

[1032] "Image enhancement" refers to digital processing to improve the quality of photographs, particularly color conversion and focus correction.

[1033] The "means for converting to color" is a technique for estimating the color of each pixel in a monochrome photograph and converting it into a color photograph with natural color tones.

[1034] "Focus correction and sharpening" is a technique that detects blurry areas in an image and applies a sharpening filter to make the photo clearer.

[1035] "Means for analyzing emotions from the user's voice and facial expressions" refers to software or algorithms that use a microphone or camera to collect data on the user's voice and facial expressions and analyze the emotions.

[1036] "Means for interacting with users using generative AI" refers to a system that utilizes artificial intelligence to interact with users and receive their instructions.

[1037] The "means for photo customization" is a system for photo editing, such as color adjustment and removal of unwanted background, based on user instructions and emotional analysis.

[1038] The "means for providing customized photos to users" is a mechanism for delivering the final processed photo data to users.

[1039] The present invention provides a system that colorizes old monochrome photos, corrects the focus, and makes them clearer, as well as analyzes the user's emotions and customizes the photos based on those emotions. This system is composed of a server and a terminal, and includes the following means:

[1040] First, the user prepares an old monochrome photograph and takes or scans it using a device (smartphone or tablet). At this time, the device displays a shooting guide that instructs the appropriate shooting angle and lighting conditions. The captured photograph is previewed, and if the user is satisfied with it, it is sent to the server.

[1041] The server temporarily stores the received photo data and verifies its integrity. It then runs an image correction algorithm to convert the monochrome photo to color, using image correction technologies such as OpenCV and TensorFlow. The colorized photo is then passed to a focus correction algorithm to sharpen any blurry areas.

[1042] The server then launches an emotion engine, using the camera and microphone to analyze the user's voice and facial expression data in real time. It performs emotion analysis using Google Cloud's Natural Language API. After determining the user's emotion, it uses a generative AI model (e.g., OpenAI's GPT-4) to generate a dialogue interface for the user. Through this interface, the user can provide specific customization instructions, such as "blur the background" or "change the color of the outfit to blue."

[1043] The emotion engine analyzes the user's emotions in real time and provides feedback to the generative AI, which then customizes the app based on the user's emotions. For example, if the user is using a wedding photo and the "joy" emotion is detected, the generative AI will suggest a warm-toned background.

[1044] As a concrete example, when colorizing and customizing wedding photos, if a user requests that the background be "warm toned," the following prompt sentence can be used for the generative AI model:

[1045] Example prompt sentence:

[1046] The user emotion is "joy." The provided photo is an old black-and-white wedding photo that has been colorized and focused. Create an ad using this photo as a base, suggesting a warm-toned background and an elegant tagline that will inspire users.

[1047] The server then applies the instructions and delivers the final, customized photo to the user. The photo data is then saved and a download link is provided to the user via email or in-app notification. The user can then download and use the final photo data via this link.

[1048] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1049] Step 1:

[1050] The user prepares an old black-and-white photograph and turns on the device (smartphone or tablet). The device displays a shooting guide, instructing on the appropriate shooting angle and lighting conditions. The user takes a photo and checks the preview. The user selects a photo they are satisfied with and presses the "upload" button in the dedicated app. At this point, the input is the digital data of the black-and-white photograph, and the output is photo data ready to be sent to the server.

[1051] Step 2:

[1052] The device compresses the selected photo data and sends it to the server. The input is the photo data stored on the device, and the output is the compressed photo data sent to the server. The server verifies the integrity of the received photo data and checks for omissions or errors.

[1053] Step 3:

[1054] The server runs image correction AI to convert received monochrome photos to color. It uses image correction technologies such as OpenCV and TensorFlow to estimate the color of each pixel and generate natural color tones. The input is monochrome photo data, and the output is colorized photo data.

[1055] Step 4:

[1056] The server passes the colorized photo data to a focus correction algorithm, which detects blurry areas in the image and applies a sharpening filter. The input is the colorized photo data, and the output is the sharpened photo data.

[1057] Step 5:

[1058] The server activates the emotion engine and uses the device's camera and microphone to analyze the user's voice and facial expressions. The device sends the user's voice and facial expression data to the emotion engine in real time. The input is the user's voice and facial expression data, and the output is the analyzed emotion data.

[1059] Step 6:

[1060] The server receives the judgment results from the emotion engine and connects with a generative AI model (e.g., OpenAI's GPT-4). The generative AI model generates a dialogue interface based on the user's emotions and instructions, and makes specific customization suggestions to the user. The input is emotion data and user instructions, and the output is customization suggestions.

[1061] Step 7:

[1062] The user provides customization instructions through a dialogue interface. The generative AI model analyzes the instructions and applies modifications to the initially processed photo. The input is the user's customization instructions, and the output is customized photo data.

[1063] Step 8:

[1064] The emotion engine analyzes the user's emotions in real time and provides feedback to the generative AI. The generative AI model then suggests additional modifications based on the emotions, resulting in optimal customization. The input is emotion data and user feedback, and the output is further modified photo data.

[1065] Step 9:

[1066] The server generates the final edited photo data and creates a permanent link. The server saves the final photo data and provides the user with a download link via email or in-app notification. The input is the final customized photo data, and the output is the download link.

[1067] Step 10:

[1068] The user clicks the provided link and downloads the final photo data through a dedicated app or browser. The user saves the downloaded photos to their device and prints them out or shares them with family and friends as needed. The input is the download link and the output is the downloaded photo data.

[1069] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1070] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1071] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1072] [Third embodiment]

[1073] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1074] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1075] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1076] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1077] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1078] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1079] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1080] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1081] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1082] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1083] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1084] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1085] The following describes in detail an embodiment of the present invention. The present invention is a system for converting old black-and-white photographs to color and sharpening out-of-focus photographs. This system is implemented through a dedicated app that is easy for users to operate, and includes the following processing flow:

[1086] Taking and uploading photos

[1087] 1. The user prepares an old black-and-white photo and launches the dedicated app.

[1088] 2. The device (smartphone or tablet) starts the camera and the user takes a photo. A shooting guide is displayed, instructing the user on the appropriate shooting angle and lighting conditions.

[1089] 3. The user can preview the photo they have taken and retake it if necessary.

[1090] 4. The user selects the photo they are satisfied with and presses the "Upload" button in the dedicated app.

[1091] 5. The device compresses the selected photo data and sends it to the server.

[1092] AI-based initial processing

[1093] 1. The server receives the photo data and runs the image correction AI. First, it converts the monochrome photo to color. This process involves an algorithm that colors each pixel to create natural-looking tones.

[1094] 2. A focus correction algorithm is then used to detect blurry areas in the colorized photo and a sharpening filter is applied, resulting in a clearer image.

[1095] Customization Process

[1096] 1. The server uses the generative AI model to generate a dialogue interface with the user, through which the user can provide specific customization instructions.

[1097] 2. Users can interact with the AI ​​to adjust the colors to their liking or remove unnecessary background elements. For example, they can communicate specific requests such as "I want the background blurred" or "I want the costume color changed to blue."

[1098] 3. The generative AI model analyzes the user's instructions and applies real-time corrections to the initially processed photo.

[1099] 4. The server displays a preview of the edited result to the user and asks for confirmation. The user can make further adjustments while viewing the preview.

[1100] Final data storage and distribution

[1101] 1. The server generates the final edited photo data and creates a permanent link.

[1102] 2. The server saves the final photo data and provides the user with a download link via email or in-app notification.

[1103] 3. The user clicks on the provided link and downloads the final photo data via a dedicated app or browser.

[1104] 4. Users can save the downloaded photos to their own devices and print them out or share them with family and friends as needed.

[1105] Specific examples

[1106] Example 1: Colorizing a black and white wedding photo

[1107] 1. The user launches the app and takes an old black-and-white wedding photo with their camera.

[1108] 2. The device sends the captured image data to the server.

[1109] 3. The server colorizes and sharpens the received photo.

[1110] 4. The server interacts with the user through the generated AI model and receives instructions on adjusting the background.

[1111] 5. The server generates the final retouched photo and provides the user with a download link.

[1112] 6. User downloads and saves the final photo via the link.

[1113] Example 2: Sharpening an out-of-focus photo of grandparents

[1114] 1. The user digitizes old photos using a scanner and uploads them using a dedicated app.

[1115] 2. The device sends the scanned data to the server.

[1116] 3. The server receives the data and performs focus correction and colorization processing.

[1117] 4. The server interacts with the generated AI model and modifies facial details based on the user's wishes.

[1118] 5. The server generates the final photo after user confirmation and provides a download link.

[1119] 6. Users download, save and share.

[1120] By implementing this invention, old black-and-white photographs can be easily colorized and out-of-focus photographs can be sharpened, allowing users to vividly relive past memories. This makes the system extremely useful, especially for people in their 50s and older.

[1121] The processing flow will be explained below.

[1122] Step 1:

[1123] The user prepares an old black-and-white photo and launches the dedicated app.

[1124] Step 2:

[1125] The device (smartphone or tablet) starts the camera and the user takes a photo. A shooting guide is displayed, instructing the appropriate shooting angle and lighting conditions.

[1126] Step 3:

[1127] The user can preview the photo they have taken and take another photo if necessary.

[1128] Step 4:

[1129] The user selects the photo they are satisfied with and presses the "Upload" button in the dedicated app.

[1130] Step 5:

[1131] The terminal compresses the selected photo data and transmits it to the server.

[1132] Step 6:

[1133] The server receives the photo data and temporarily stores it in an input database, where it checks the data for consistency and for missing or error information.

[1134] Step 7:

[1135] The server runs an image correction AI to convert the received monochrome photo into color, which then colors each pixel to produce a natural color tone.

[1136] Step 8:

[1137] The server passes the colorized photo to a focus correction algorithm, which detects blurry areas and applies a sharpening filter.

[1138] Step 9:

[1139] The server sends the color photo data after initial processing to the generative AI model.

[1140] Step 10:

[1141] The server generates an interactive interface with the user through the generated AI model.

[1142] Step 11:

[1143] The user can use the dialogue interface to make color adjustments or remove unnecessary backgrounds, for example, by inputting instructions such as "I want the background to be blurred" or "I want the color of the costumes to be changed to blue."

[1144] Step 12:

[1145] The generative AI model analyzes the user's instructions and applies corrections to the initially processed photo.

[1146] Step 13:

[1147] The server displays the correction results in real time and asks the user for confirmation, allowing the user to give further instructions while viewing the preview and adjust the final result.

[1148] Step 14:

[1149] The server generates the final edited photo data and creates a permanent link.

[1150] Step 15:

[1151] The server stores the final photo data and provides the user with a download link via email or in-app notification.

[1152] Step 16:

[1153] The user clicks on the provided link and downloads the final photo data via a dedicated app or browser.

[1154] Step 17:

[1155] Users can save the downloaded photos to their own devices and print them out or share them with family and friends as needed.

[1156] Example 1

[1157] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1158] Conventional photo correction systems often require manual work, whether it's converting old black-and-white photos to color or correcting out-of-focus photos to make them clearer, making the process extremely time-consuming. Furthermore, specific corrections and customizations require users to have advanced skills, making them difficult for average users without specialized knowledge. For this reason, there is a demand for a system that allows users to easily perform high-quality photo corrections.

[1159] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1160] In this invention, the server includes a device that takes or scans old black-and-white photographs, a device that transmits the taken or scanned black-and-white photograph data to the server, a program that performs image correction on the server and converts the black-and-white photographs to color, a program that corrects the focus of the colorized photographs to make them clearer, a program that interacts with the user using a generative AI model and customizes the photographs based on the user's instructions, and a device that finally provides the customized photographs to the user. This allows users to easily perform high-quality photo correction without specialized knowledge.

[1161] An "old black and white photograph" is a black and white (monochrome) photograph taken in the past, and refers to an image that does not contain color information.

[1162] A "photographing or scanning device" is a device for converting a physical photograph into a digital image, and includes hardware devices such as a camera or scanner.

[1163] "Transmitting device" refers to a hardware or software device for transferring photographed or scanned digital image data to a server via a network.

[1164] "Server" refers to a remote computer or cloud-based computing system for receiving and processing digital image data.

[1165] "Image correction program" refers to a software application that performs operations such as adjusting color tone and correcting exposure on digital images.

[1166] A "color conversion program" refers to a software algorithm that adds color information to each pixel of a monochrome image to convert it into a natural color image.

[1167] A "focus and sharpening program" refers to software that detects blurry areas in an image and applies a sharpening filter to improve the image.

[1168] "Generative AI model" refers to an artificial intelligence model that interacts with users, analyzes their instructions based on natural language prompts, and is responsible for making corrections and improvements to images.

[1169] "Customization program" refers to software that modifies images based on a user's specific needs.

[1170] The "providing device" refers to a device for distributing the final edited image data to the user, and includes email, cloud storage services, etc.

[1171] The system of this invention is mainly designed to convert old black and white photos into color and sharpen out-of-focus photos, allowing users to vividly relive their past memories. The whole system consists of three main components: the user, the terminal, and the server.

[1172] First, the user prepares an old black-and-white photograph and launches the dedicated application. This application runs on devices such as smartphones and tablets, and activates the camera to take a photo. Shooting guides are displayed, allowing the user to take a photo with the optimal lighting and angle.

[1173] The device then displays a preview of the captured photo, allowing the user to review and retake the photo until satisfied. Once a satisfactory photo is selected, the device compresses the photo data and sends it to the server.

[1174] The server receives the photo data and performs image correction. The image correction process first converts the monochrome photo to color. Specifically, it uses Python's OpenCV library to apply an algorithm that estimates the appropriate color for each pixel and generates natural-looking tones. Next, a focus correction algorithm using TensorFlow is applied, detecting blurred areas and applying a sharpening filter to make the entire photo clearer.

[1175] In addition, a customization process is provided in which users interact with a generative AI model. This generative AI model uses advanced natural language processing techniques, such as GPT-4, to modify photos based on prompts entered by the user. Users can provide specific customization instructions through the dialogue interface, such as "blur the background" or "change the color of the outfit to blue." The generative AI model analyzes these instructions and applies them to the photo in real time.

[1176] The final customized photo is generated as high-resolution image data by the server. The server stores this data in cloud storage and generates a permanent link that is provided to the user. The user receives a download link via email or in-app notification, and can download the final photo data through a dedicated application or browser and save it to their device. This allows the user to print out the final photo or share it with family and friends.

[1177] As a concrete example, we will show the process of colorizing a black-and-white wedding photo. The user launches a dedicated app and takes a photo of an old black-and-white wedding photo with their camera. The device sends the captured data to a server, which then colorizes and sharpens the received photo. The background is then adjusted based on the user's instructions via a generative AI model, and the user is provided with a download link for the final edited photo.

[1178] By implementing this invention, users can easily colorize old monochrome photographs and sharpen out-of-focus photographs without any specialized knowledge. This provides a very useful system for vividly reviving past memories, especially for those in their 50s and older.

[1179] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1180] Step 1:

[1181] The user prepares an old black-and-white photo and launches the dedicated app.

[1182] Input: Physical black and white photo

[1183] Output: Launching the dedicated app

[1184] Specific operation: The user prepares a physical black-and-white photo and launches a dedicated app on their smartphone or tablet.

[1185] Step 2:

[1186] The device will activate the camera and take a photo.

[1187] Input: User-initiated camera activation and shooting instructions

[1188] Output: Digitized black and white photo capture

[1189] Specific operation: The device's camera is activated, a shooting guide is displayed, and the user presses the shooting button to digitize a monochrome photo.

[1190] Step 3:

[1191] The device displays a preview of the photo for the user to review and retake.

[1192] Input: Digitized black and white photograph

[1193] Output: Show preview image

[1194] Specific operation: The device displays a preview of the photo taken and displays the "Confirm" and "Retake" buttons. The user checks the preview and presses the "Retake" button if they need to take another photo.

[1195] Step 4:

[1196] The user selects a photo they are satisfied with and presses the upload button.

[1197] Input: A preview image selected by the user.

[1198] Output: Ready to upload status

[1199] Specific operation: The user presses the "Confirm" button, which takes them to the upload screen of the dedicated app. The user presses the "Upload" button.

[1200] Step 5:

[1201] The terminal compresses the selected photo data and transmits it to the server.

[1202] Input: A digitized black and white photograph selected by the user

[1203] Output: Compressed photo data sent to the server

[1204] Specific operation: The device compresses the image data and sends it to a server via the Internet.

[1205] Step 6:

[1206] The server receives the photo data and performs image correction.

[1207] Input: Compressed photo data

[1208] Output: Colorized and focused image

[1209] What it does: The server receives the photo data and colorizes the monochrome photo using Python's OpenCV library, then uses TensorFlow to apply a focus correction algorithm to make the photo clearer.

[1210] Step 7:

[1211] The server uses the generative AI model to generate a dialogue interface with the user.

[1212] Input: Colorized and focus-corrected image data

[1213] Output: Generate a conversational interface

[1214] Specific operation: The server uses a generative AI model (e.g., GPT-4) to generate a dialogue interface and provides a screen for the user to input customization instructions.

[1215] Step 8:

[1216] The user gives specific customization instructions.

[1217] Input: Prompt text given by the user (e.g., "I want the background to be blurred" or "I want the color of the costume to be blue")

[1218] Output: Instruction analysis results by the generative AI model

[1219] Specific operation: The user inputs a prompt sentence through the dialogue interface, and the AI ​​receives and analyzes it.

[1220] Step 9:

[1221] A generative AI model modifies photos based on user instructions.

[1222] Input: Prompt statement analysis result

[1223] Output: Corrected image data

[1224] How it works: The generative AI model modifies specific parts of the image based on user instructions and makes corresponding adjustments in real time.

[1225] Step 10:

[1226] The server displays a preview of the correction results to the user.

[1227] Input: Corrected image data

[1228] Output: Preview screen display

[1229] What happens: The server previews the corrected image to the user, allowing the user to make further adjustments.

[1230] Step 11:

[1231] The server generates the final edited photo data and creates a fixed link.

[1232] Input: Final corrected image data

[1233] Output: Cloud storage of image data and generation of a permanent link

[1234] Specific operation: The server generates a high-resolution image after correction is completed, saves it to cloud storage, and then creates a permanent link.

[1235] Step 12:

[1236] The server notifies the link and provides it to the user.

[1237] Input:permalink

[1238] Output: User notification (email or in-app notification)

[1239] Specific operation: The server notifies the user of the fixed link and provides a download link.

[1240] Step 13:

[1241] The user clicks on the provided link to download the final photo data.

[1242] Input:permalink

[1243] Output: Final downloaded photo data

[1244] Specific operation: The user clicks on the link and downloads the final photo data via a dedicated app or web browser.

[1245] Step 14:

[1246] Save the photos you download to your device and share them as needed.

[1247] Input: Downloaded photo data

[1248] Output: Photos stored on the device and shared photos

[1249] What it does: Users can save the downloaded photos to their devices, share them with family and friends, and even print them out.

[1250] (Application example 1)

[1251] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1252] In recent years, advances in digital technology have led to an increasing number of users wanting to digitize and preserve old photographs. However, converting old black-and-white photographs to color and sharpening blurry photographs is a time-consuming process that requires specialized knowledge. Furthermore, there are limited means to easily provide such services, especially in brick-and-mortar stores. This makes it difficult for customers to restore photos at their own convenience, leading to lower service satisfaction. Therefore, there is a need for a system that can easily colorize and sharpen old photographs in-store.

[1253] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1254] In this invention, the server includes means for taking or scanning old monochrome photographs, means for transmitting the taken or scanned monochrome photograph data to the server, means for performing image correction on the server to convert the monochrome photograph to color, means for correcting the focus of the colorized photograph to make it clearer, means for interacting with the user using a generative AI and customizing the photograph based on the user's instructions, means for providing the user with the final customized photograph, and means for the user to use the application that performs the photo customization process in a physical store environment. This allows customers to easily colorize and sharpen old photographs in a physical store.

[1255] An "old black-and-white photograph" is a photograph taken in the past that lacks color and is expressed in black and white rather than color.

[1256] "Means of photographing or scanning" refers to the use of a device such as a camera or scanner to obtain a photograph as digital data.

[1257] A "transmitting means" is a communication means for sending digital data over the Internet or a local network to a remote server.

[1258] A "server" is a computer system that receives and processes digital data.

[1259] "Image correction means" refers to software or algorithms for adjusting contrast and correcting color of digital images.

[1260] "Methods for converting to color" are techniques for adding color to monochrome photographs to reproduce them as natural color photographs.

[1261] "Focus correction and sharpening" is a method for sharpening blurry photos and improving the clarity of details.

[1262] "Generative AI" is a system that uses an artificial intelligence model to interact with users in natural language and process images based on their explanations.

[1263] "Means for interacting with users and customizing" refers to technology that uses AI to understand users' requests and modify or process specific parts of images.

[1264] A "means for providing a customized photograph to a user" is a method by which the final, enhanced digital image can be downloaded or viewed by the user.

[1265] "Methods available to users in a physical store environment" refers to the use of applications and digital devices by customers to receive services within a physical store.

[1266] The following describes in detail the mode for carrying out the invention. This system has various functions for converting old black-and-white photos to color and sharpening out-of-focus photos. It is intended for use in brick-and-mortar stores, and customers can easily use these functions using a dedicated application.

[1267] Hardware and software used

[1268] The system uses the following hardware and software:

[1269] Hardware:

[1270] Devices (smartphones, tablets, cameras)

[1271] Server (high performance processor, internet connection)

[1272] software:

[1273] OpenCV (image capture, image processing)

[1274] PIL (Python Imaging Library) (Image Processing)

[1275] requests (HTTP communication library)

[1276] Generative AI model (user interaction, image customization)

[1277] Processing flow explanation

[1278] Taking and uploading photos

[1279] The user launches a dedicated app and prepares an old black-and-white photo. The device then activates the camera, and the user takes a photo. A shooting guide is displayed, instructing on the appropriate shooting angle and lighting conditions. The user can check a preview of the photo they have taken and take another photo if necessary. When the user selects a photo they are satisfied with and presses the upload button, the device compresses the photo data and sends it to the server.

[1280] AI-based initial processing

[1281] The server receives the photo data and runs an image correction AI, first converting the black and white photo to color, then using a focus correction algorithm to detect blurry areas in the colorized photo and applying a sharpening filter.

[1282] Customization Process

[1283] The server uses the generative AI model to generate a dialogue interface for the user. The user can use this interface to give specific customization instructions. For example, they can express specific requests such as "I want the background blurred" or "I want the costume color changed to blue." The generative AI model analyzes the user's instructions and applies real-time corrections to the initially processed photo. The server then displays a preview of the corrections to the user and receives confirmation and additional instructions.

[1284] Final data storage and distribution

[1285] The server generates the final edited photo data and creates a permanent link. The server saves the final photo data and provides the user with a download link via email or in-app notification. The user clicks the link and downloads the final photo data via a dedicated app or browser. The user saves the downloaded photo to their device and can print it out or share it with family and friends as needed.

[1286] Examples of concrete examples and prompts

[1287] As a concrete example, customers can take old family photos with the app in a physical store where their families gather, and have them colorized and sharpened on the spot. This service allows customers to vividly relive their past memories and share them as new memories.

[1288] Example prompt sentence:

[1289] 1. Take a photo or upload an existing one.

[1290] 2. Sending photos to server... Please wait.

[1291] 3. Processing the photo… We will correct the focus and add color to the original photo.

[1292] 4. Processing completed! Please check.

[1293] 5. Do you want to customize further? For example, "I want the background to be blurred" or "I want the costume color to be changed to blue."

[1294] This system allows customers to easily colorize and sharpen old photos in-store.

[1295] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1296] Step 1:

[1297] Photographing and scanning

[1298] The user prepares an old black-and-white photo and launches a dedicated app. The device launches the camera and the user takes a photo. A shooting guide is displayed, instructing on the appropriate shooting angle and lighting conditions. The input is the old black-and-white photo, and the output is a digital copy of the photo. A preview is provided to confirm that the appropriate shooting conditions are met, allowing the user to decide whether or not to retake the photo.

[1299] Step 2:

[1300] Upload

[1301] When the user selects a photo they are satisfied with and presses the upload button, the device compresses the photo data and sends it to the server. The input is a digital photo, and the output is compressed photo data. This process improves data transfer speed and reduces server load.

[1302] Step 3:

[1303] Initial Data Processing

[1304] The server receives the photo data and first runs the image correction AI. The input is compressed photo data, and the output is colorized photo data. Specifically, it applies an algorithm that adds color to each pixel of the monochrome photo and generates natural-looking tones.

[1305] Step 4:

[1306] Focus Correction

[1307] The server then uses a focus correction algorithm to detect blurred areas in the colorized photo and applies a sharpening filter. The input is the colorized photo data, and the output is the sharpened photo data. Specifically, the server runs a filter to improve the contrast and detail in the blurred areas.

[1308] Step 5:

[1309] Customization with generative AI

[1310] The server uses the generative AI model to generate a dialogue interface for the user. The user can then provide specific customization instructions through this interface. The input is the user's text-based instructions, and the output is customized photo data. Specific operations include analyzing requests such as "blur the background" or "change the color of the costume to blue," and reflecting the results in image processing.

[1311] Step 6:

[1312] Preview and fix

[1313] The server displays a preview of the correction results to the user and receives confirmation and additional instructions from the user. The input is customized photo data, and the output is the final corrected photo data. Specifically, it provides an interface that allows the user to make further adjustments while viewing the preview.

[1314] Step 7:

[1315] Final data storage and distribution

[1316] The server generates the final corrected photo data and creates a permanent link. The server saves the final photo data and provides the user with a download link via email or in-app notification. The input is the final corrected photo data and the output is a permanent link. Specifically, this involves saving the photo in a database and generating a link to provide it to the user.

[1317] Through this process, customers can easily colorize and sharpen old photos in-store.

[1318] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1319] The following describes in detail the embodiments of the present invention. The present invention combines a system for colorizing old black-and-white photos and sharpening out-of-focus photos with an emotion engine that recognizes the user's emotions. This system is implemented through a dedicated app that is easy for users to operate, and includes the following processing flow:

[1320] Taking and uploading photos

[1321] 1. The user prepares an old black-and-white photo and launches the dedicated app.

[1322] 2. The device (smartphone or tablet) starts the camera and the user takes a photo. A shooting guide is displayed, instructing the user on the appropriate shooting angle and lighting conditions.

[1323] 3. The user can preview the photo they have taken and retake it if necessary.

[1324] 4. The user selects the photo they are satisfied with and presses the "Upload" button in the dedicated app.

[1325] 5. The device compresses the selected photo data and sends it to the server.

[1326] AI-based initial processing

[1327] 1. The server receives the photo data and temporarily stores it in an input database, where it checks the data for consistency and omissions or errors.

[1328] 2. The server runs the image correction AI to convert the received monochrome photo to color. The image correction AI colors each pixel to produce natural-looking colors.

[1329] 3. The server passes the colorized photo to a focus correction algorithm, which detects blurry areas and applies a sharpening filter.

[1330] Emotion engine processing

[1331] 1. The server activates the emotion engine and uses the camera and microphone to analyze the user's voice and facial expressions.

[1332] 2. The device transmits the user's voice and facial expression data to the emotion engine in real time.

[1333] 3. The emotion engine analyzes the received data and determines the user's emotion. For example, if the user is smiling, it is determined to be "joy," and if they are frowning, it is determined to be "confusion."

[1334] 4. The server receives the judgment results from the emotion engine and works with the generation AI.

[1335] Customization Process

[1336] 1. The server uses the generative AI model to generate a dialogue interface with the user.

[1337] 2. The user enters specific customization instructions through the dialogue interface, such as "I want the background to be blurred" or "I want the color of the costume to be blue."

[1338] 3. The generative AI model analyzes the user's instructions and applies corrections to the initially processed photo.

[1339] 4. The emotion engine analyzes the user's emotions in real time and provides feedback to the generative AI, allowing the generative AI model to suggest additional modifications based on the emotions.

[1340] 5. The server displays the edited results in real time and asks the user for confirmation. The user can make further adjustments while viewing the preview.

[1341] Final data storage and distribution

[1342] 1. The server generates the final edited photo data and creates a permanent link.

[1343] 2. The server saves the final photo data and provides the user with a download link via email or in-app notification.

[1344] 3. The user clicks on the provided link and downloads the final photo data via a dedicated app or browser.

[1345] 4. Users can save the downloaded photos to their own devices and print them out or share them with family and friends as needed.

[1346] Specific examples

[1347] Example 1: Colorization of Black and White Wedding Photos and Emotion-Based Customization

[1348] 1. The user launches the app and takes an old black-and-white wedding photo with their camera.

[1349] 2. The device sends the captured image data to the server.

[1350] 3. The server colorizes and sharpens the received photo.

[1351] 4. The server interacts with the user through the generative AI model and emotion engine and receives instructions on adjusting the background.

[1352] 5. The emotion engine detects the user's happiness and suggests warmer background colors.

[1353] 6. The server generates the final retouched photo and provides the user with a download link.

[1354] 7. User downloads and saves the final photo via the link.

[1355] Example 2: Sharpening out-of-focus photos of grandparents and emotion-based customization

[1356] 1. The user digitizes old photos using a scanner and uploads them using a dedicated app.

[1357] 2. The device sends the scanned data to the server.

[1358] 3. The server receives the data and performs focus correction and colorization processing.

[1359] 4. The server interacts with the user through the generative AI model and emotion engine, and performs facial detail correction.

[1360] 5. The emotion engine analyzes the user's emotions, and if it detects "confusion," the generative AI will suggest more detailed adjustment methods.

[1361] 6. The server generates the final photo after user confirmation and provides a download link.

[1362] 7. Users can download, save, or share.

[1363] By implementing this invention, not only can old black-and-white photographs be easily colorized and out-of-focus photographs be made clearer, but the system can also be optimally customized based on the user's emotions, allowing users to relive their past memories in a richer way. This makes the system extremely useful, especially for those in their 50s and older.

[1364] The processing flow will be explained below.

[1365] Step 1:

[1366] The user prepares an old black-and-white photo and launches the dedicated app.

[1367] Step 2:

[1368] The device (smartphone or tablet) starts the camera and the user takes a photo. A shooting guide is displayed, instructing the appropriate shooting angle and lighting conditions.

[1369] Step 3:

[1370] The user can preview the photo they have taken and take another photo if necessary.

[1371] Step 4:

[1372] The user selects the photo they are satisfied with and presses the "Upload" button in the dedicated app.

[1373] Step 5:

[1374] The terminal compresses the selected photo data and transmits it to the server.

[1375] Step 6:

[1376] The server receives the photo data and temporarily stores it in an input database, where it checks the data for consistency and for missing or error information.

[1377] Step 7:

[1378] The server runs an image correction AI to convert the received monochrome photo into color, which then colors each pixel to produce a natural color tone.

[1379] Step 8:

[1380] The server passes the colorized photo to a focus correction algorithm, which detects blurry areas and applies a sharpening filter.

[1381] Step 9:

[1382] The server runs the emotion engine and uses the camera and microphone to analyze the user's voice and facial expressions.

[1383] Step 10:

[1384] The device transmits the user's voice and facial expression data to the emotion engine in real time.

[1385] Step 11:

[1386] The emotion engine analyzes the received data and determines the user's emotion. For example, if the user is smiling, it is determined to be "joy," and if they are frowning, it is determined to be "confusion."

[1387] Step 12:

[1388] The server receives the judgment results from the emotion engine and works in conjunction with the generation AI.

[1389] Step 13:

[1390] The server uses the generative AI model to generate a dialogue interface with the user.

[1391] Step 14:

[1392] The user enters specific customization instructions through a dialogue interface, such as "I want the background to be blurred" or "I want the color of the costume to be changed to blue."

[1393] Step 15:

[1394] The generative AI model analyzes the user's instructions and applies corrections to the initially processed photo.

[1395] Step 16:

[1396] The emotion engine analyzes the user's emotions in real time and provides feedback to the generative AI.

[1397] Step 17:

[1398] The generative AI model suggests additional modifications based on the user's sentiment.

[1399] Step 18:

[1400] The server displays the correction results in real time and asks the user for confirmation, allowing the user to make further adjustments while viewing the preview.

[1401] Step 19:

[1402] The server generates the final edited photo data and creates a permanent link.

[1403] Step 20:

[1404] The server stores the final photo data and provides the user with a download link via email or in-app notification.

[1405] Step 21:

[1406] The user clicks on the provided link and downloads the final photo data via a dedicated app or browser.

[1407] Step 22:

[1408] Users can save the downloaded photos to their own devices and print them out or share them with family and friends as needed.

[1409] Example 2

[1410] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1411] Conventional photo restoration systems provide methods for colorizing old monochrome photographs and correcting focus, but do not offer customization that takes into account the user's emotions. As a result, it takes a lot of time and effort to achieve results that satisfy the user. The purpose of this invention is to solve these problems and provide a system that is easy for users to operate and that allows customization based on emotions.

[1412] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1413] In this invention, the server includes means for taking or scanning old monochrome photographs, means for transmitting the taken or scanned monochrome photograph data to an information processing device, means for performing image correction on the information processing device to convert the monochrome photograph to color, means for correcting the focus of the colorized photograph to make it clearer, means for analyzing the user's voice and facial expression in real time to determine their emotions, means for interacting with the user using a generative AI and customizing the photo based on the user's instructions and the determined emotions, and means for finally providing the customized photo to the user. This enables photo restoration and customization that satisfies the user to be performed quickly and easily.

[1414] "Old monochrome photographs" are black and white photographic data that were taken in the past and do not contain color information.

[1415] "Photographing or scanning means" refers to methods and devices that use cameras or scanners to convert physical photographs or images into digital data.

[1416] "Information processing device" refers to a computer system such as a server or cloud computing system for processing captured or scanned image data.

[1417] "Method of image correction and converting black-and-white photos to color" refers to an image processing technology that assigns an appropriate color to each pixel of a black-and-white photo to reproduce natural color tones.

[1418] "Method for correcting focus and sharpening colorized photographs" refers to a technique that detects blurry areas of an image and applies a sharpening filter to them to increase the overall clarity of the image.

[1419] "Means for analyzing a user's voice and facial expressions in real time to determine emotions" refers to a system that uses a microphone or camera to collect a user's voice and facial expressions, and analyzes that data to identify the user's emotional state.

[1420] "Generative AI" refers to an artificial intelligence system that interacts with users based on input data and instructions and automatically performs the necessary customizations.

[1421] "Means for interacting with a user and customizing a photo based on the user's instructions and determined emotions" refers to a method and apparatus that receives instructions from a user via a user interface and further processes or modifies a photo taking into account the user's emotional state.

[1422] "Means for providing the final customized photo to the user" means a system that provides the final processed photo data to the user via a download link or in-app notification.

[1423] This system aims to restore a clear image by colorizing old monochrome photographs and adjusting the focus. It also analyzes the user's voice and facial expressions to determine their emotions, and customizes the photo restoration process based on those emotions to ensure the user is satisfied with the results. This system is implemented using the following hardware and software:

[1424] Hardware configuration:

[1425] Device: A device such as a smartphone, tablet, or computer that contains a camera and microphone and collects user input and emotional data.

[1426] Server: A server with high-performance processing power, where image processing, database management, and AI model execution take place.

[1427] Software configuration:

[1428] Image Enhancement Software: This includes AI algorithms that colorize old black-and-white photos and correct focus. Specific examples of the software used include OpenCV and TensorFlow.

[1429] Emotion engine: Software that analyzes the user's voice and facial expressions to determine their emotions. Specific software used includes "DeepFace" and "Google Cloud Speech-to-Text."

[1430] Generative AI models: Personalize photos based on user input and emotions, using natural language processing and generative models like GPT-3 and DALL·E.

[1431] Examples:

[1432] 1. Colorization of monochrome wedding photos and emotion-based customization

[1433] The user launches the dedicated app and takes an old black-and-white wedding photo with their camera.

[1434] The device sends the captured image data to the server.

[1435] The server then colorizes and sharpens the received photographs.

[1436] The server interacts with the user through the generative AI model and emotion engine, and receives instructions for adjusting the background.

[1437] The emotion engine detects the user's happiness and suggests warmer background colors.

[1438] The server generates the final retouched photo and provides the user with a download link.

[1439] The user downloads and saves the final photo via a link.

[1440] 2. Sharpening out-of-focus photos of grandparents and emotion-based customization

[1441] Users digitize old photos using a scanner and upload them using a dedicated app.

[1442] The device sends the scanned data to the server.

[1443] The server receives the data and performs focus correction and colorization processing.

[1444] The server interacts with the user through the generative AI model and emotion engine, and performs facial detail correction.

[1445] The emotion engine analyzes the user's emotions, and if it detects "confusion," the generative AI will suggest more detailed adjustment methods.

[1446] The server generates the final photo after user confirmation and provides a download link.

[1447] Users can download, save and share.

[1448] Example prompt sentence:

[1449] 1. "Blur the background in this photo to make the person stand out."

[1450] 2. "Please change the color of the outfit in this photo to blue to balance the overall look."

[1451] 3. "I'd like to see a bit more sharpness in this photo to bring out the details."

[1452] This system allows users to easily colorize old black-and-white photos, improve their clarity, and even customize them individually, allowing users to more vividly relive their rich memories.

[1453] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1454] Step 1:

[1455] The user prepares an old black-and-white photo and launches the dedicated app.

[1456] Input: Old black and white photo

[1457] Output: Ready to be photographed or scanned

[1458] Step 2:

[1459] When the user presses the "Take Photo" button on the dedicated app, the device's camera will launch and a photo guide will be displayed.

[1460] Input: User action (pressing the "Take Photo" button)

[1461] Output: Starts the camera and displays the shooting guide

[1462] What it does: The shooting guide displays messages instructing you on the appropriate shooting angles and lighting conditions.

[1463] Step 3:

[1464] The user takes a photo using the camera and then sees a preview screen after taking the photo, which allows them to choose whether to retake the photo or use the one they just took.

[1465] Input: Photo data taken

[1466] Output: Preview screen and user choice (retake or use)

[1467] Step 4:

[1468] The user selects a photo they are satisfied with and presses the "Upload" button in the dedicated app.

[1469] Input: User operation (pressing the "Upload" button)

[1470] Output: Photo ready to upload

[1471] Step 5:

[1472] The device compresses the selected photo data and sends it to the server. The device uses a compression algorithm to minimize the data size and reduce transmission time.

[1473] Input: Photo data taken

[1474] Output: Compressed photo data, sent to server

[1475] Step 6:

[1476] The server receives the photo data and temporarily stores it in an input database, where it checks the data for consistency and omissions or errors.

[1477] Input: Compressed photo data

[1478] Output: Data stored in the database, consistency check results

[1479] Step 7:

[1480] The server runs an image correction AI to convert the received monochrome photo into color, assigning the appropriate color to each pixel and reproducing natural color tones.

[1481] Input: Monochrome photo data

[1482] Output: Colorized photo data

[1483] How it works: An AI algorithm colors each image pixel by pixel, ensuring a natural overall color tone.

[1484] Step 8:

[1485] The server passes the colorized photo through a focus correction algorithm, which detects blurry areas and applies a sharpening filter.

[1486] Input: Colorized photo data

[1487] Output: Focused photo data

[1488] What it does: Image analysis identifies blurry areas and applies a sharpening filter.

[1489] Step 9:

[1490] The server starts the emotion engine and initializes the camera and microphone to analyze the user's voice and facial expressions.

[1491] Input: User voice and facial expression data

[1492] Output: Initial setup completed

[1493] What it does: Camera and microphone settings are automatically adjusted, and voice and facial expression data is captured in real time.

[1494] Step 10:

[1495] The device transmits the user's voice and facial expression data to the emotion engine in real time.

[1496] Input: User's voice and facial expression data

[1497] Output: Data sent to the emotion engine

[1498] Step 11:

[1499] The emotion engine analyzes the received data and determines the user's emotion.

[1500] Input: Voice and facial expression data

[1501] Output: Sentiment analysis result (e.g., happy, confused)

[1502] Specific operation: The AI ​​algorithm analyzes voice tone, facial features, etc. to determine emotions.

[1503] Step 12:

[1504] The server passes the judgment results from the emotion engine to the generation AI and works in conjunction with the generation AI.

[1505] Input: Sentiment analysis results

[1506] Output: Data passed to the generation AI

[1507] Specific operation: Emotional data is provided to the generative AI, which generates customizations based on the user's emotions.

[1508] Step 13:

[1509] The server uses the generated AI model to generate a dialogue interface with the user.

[1510] Input: Generative AI model held by the server

[1511] Output: Generates a conversational interface

[1512] Specific operation: Using natural language processing, it dynamically generates an easy-to-use dialogue interface for users.

[1513] Step 14:

[1514] The user inputs specific customization instructions through the dialogue interface.

[1515] Input: User customization instructions

[1516] Output: Instruction data sent to the server

[1517] Step 15:

[1518] The generative AI model analyzes the user's instructions and applies corrections to the initially processed photo.

[1519] Input: User customization instructions and initial processed photo data

[1520] Output: Photo data with corrections

[1521] What it does: Based on user instructions, the image editing algorithm applies specific modifications.

[1522] Step 16:

[1523] The emotion engine analyzes the user's emotions in real time and provides feedback to the generative AI.

[1524] Input: Real-time user emotion data

[1525] Output: Feedback provided to the generative AI

[1526] What it does: The updated emotion data is passed to the generative AI, providing reference information for further customization.

[1527] Step 17:

[1528] The server displays the results of the modifications in real time and asks the user for confirmation, allowing the user to make further adjustments while viewing the preview.

[1529] Input: Modified photo data and user emotion data

[1530] Output: Preview screen and user adjustment instructions

[1531] Step 18:

[1532] The server generates the final edited photo data and creates a permanent link.

[1533] Input: Final edited photo data

[1534] Output: Download link

[1535] Step 19:

[1536] The server stores the final photo data and provides the user with a download link via email or in-app notification.

[1537] Input: Last modified photo data and user contact information

[1538] Output: Provide a download link

[1539] Step 20:

[1540] The user clicks on the provided link and downloads the final photo data via a dedicated app or browser.

[1541] Input: Download link

[1542] Output: Downloaded photo data

[1543] Step 21:

[1544] Users can save the downloaded photos to their own devices and print them out or share them with family and friends as needed.

[1545] Input: Downloaded photo data

[1546] Output: Saved and shared photos

[1547] (Application example 2)

[1548] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1549] Conventional photo editing systems simply colorize and correct uniform photos without considering the user's emotions, making it difficult to achieve the optimal customization desired by the user. Furthermore, while the advertising industry requires the generation of advertising materials based on user emotions, current technology has difficulty providing this efficiently and effectively. Therefore, there is a need for a system that can analyze user emotions and customize photos based on them.

[1550] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1551] In this invention, the server includes means for taking or scanning old monochrome photographs, means for transmitting the taken or scanned monochrome photograph data to the server, means for performing image correction on the server to convert the monochrome photograph to color, means for correcting the focus of the colorized photograph to make it clearer, means for analyzing emotions from the user's voice and facial expressions, means for interacting with the user using a generative AI and customizing the photo based on the user's instructions and emotional analysis, and means for finally providing the customized photo to the user. This enables optimal photo customization and generation of advertising materials that take the user's emotions into consideration.

[1552] An "old black and white photograph" is a black and white photograph taken a long time ago, and does not contain color information.

[1553] "Means for photographing or scanning" refers to a device or system for digitizing photographs using a camera, scanner, etc.

[1554] "Means for transmitting to a server" refers to a function for transferring digital data to a server via the Internet or other communications network.

[1555] "Image enhancement" refers to digital processing to improve the quality of photographs, particularly color conversion and focus correction.

[1556] The "means for converting to color" is a technique for estimating the color of each pixel in a monochrome photograph and converting it into a color photograph with natural color tones.

[1557] "Focus correction and sharpening" is a technique that detects blurry areas in an image and applies a sharpening filter to make the photo clearer.

[1558] "Means for analyzing emotions from the user's voice and facial expressions" refers to software or algorithms that use a microphone or camera to collect data on the user's voice and facial expressions and analyze the emotions.

[1559] "Means for interacting with users using generative AI" refers to a system that utilizes artificial intelligence to interact with users and receive their instructions.

[1560] The "means for photo customization" is a system for photo editing, such as color adjustment and removal of unwanted background, based on user instructions and emotional analysis.

[1561] The "means for providing customized photos to users" is a mechanism for delivering the final processed photo data to users.

[1562] The present invention provides a system that colorizes old monochrome photos, corrects the focus, and makes them clearer, as well as analyzes the user's emotions and customizes the photos based on those emotions. This system is composed of a server and a terminal, and includes the following means:

[1563] First, the user prepares an old monochrome photograph and takes or scans it using a device (smartphone or tablet). At this time, the device displays a shooting guide that instructs the appropriate shooting angle and lighting conditions. The captured photograph is previewed, and if the user is satisfied with it, it is sent to the server.

[1564] The server temporarily stores the received photo data and verifies its integrity. It then runs an image correction algorithm to convert the monochrome photo to color, using image correction technologies such as OpenCV and TensorFlow. The colorized photo is then passed to a focus correction algorithm to sharpen any blurry areas.

[1565] The server then launches an emotion engine, using the camera and microphone to analyze the user's voice and facial expression data in real time. It performs emotion analysis using Google Cloud's Natural Language API. After determining the user's emotion, it uses a generative AI model (e.g., OpenAI's GPT-4) to generate a dialogue interface for the user. Through this interface, the user can provide specific customization instructions, such as "blur the background" or "change the color of the outfit to blue."

[1566] The emotion engine analyzes the user's emotions in real time and provides feedback to the generative AI, which then customizes the app based on the user's emotions. For example, if the user is using a wedding photo and the "joy" emotion is detected, the generative AI will suggest a warm-toned background.

[1567] As a concrete example, when colorizing and customizing wedding photos, if a user requests that the background be "warm toned," the following prompt sentence can be used for the generative AI model:

[1568] Example prompt sentence:

[1569] The user emotion is "joy." The provided photo is an old black-and-white wedding photo that has been colorized and focused. Create an ad using this photo as a base, suggesting a warm-toned background and an elegant tagline that will inspire users.

[1570] The server then applies the instructions and delivers the final, customized photo to the user. The photo data is then saved and a download link is provided to the user via email or in-app notification. The user can then download and use the final photo data via this link.

[1571] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1572] Step 1:

[1573] The user prepares an old black-and-white photograph and turns on the device (smartphone or tablet). The device displays a shooting guide, instructing on the appropriate shooting angle and lighting conditions. The user takes a photo and checks the preview. The user selects a photo they are satisfied with and presses the "upload" button in the dedicated app. At this point, the input is the digital data of the black-and-white photograph, and the output is photo data ready to be sent to the server.

[1574] Step 2:

[1575] The device compresses the selected photo data and sends it to the server. The input is the photo data stored on the device, and the output is the compressed photo data sent to the server. The server verifies the integrity of the received photo data and checks for omissions or errors.

[1576] Step 3:

[1577] The server runs image correction AI to convert received monochrome photos to color. It uses image correction technologies such as OpenCV and TensorFlow to estimate the color of each pixel and generate natural color tones. The input is monochrome photo data, and the output is colorized photo data.

[1578] Step 4:

[1579] The server passes the colorized photo data to a focus correction algorithm, which detects blurry areas in the image and applies a sharpening filter. The input is the colorized photo data, and the output is the sharpened photo data.

[1580] Step 5:

[1581] The server activates the emotion engine and uses the device's camera and microphone to analyze the user's voice and facial expressions. The device sends the user's voice and facial expression data to the emotion engine in real time. The input is the user's voice and facial expression data, and the output is the analyzed emotion data.

[1582] Step 6:

[1583] The server receives the judgment results from the emotion engine and connects with a generative AI model (e.g., OpenAI's GPT-4). The generative AI model generates a dialogue interface based on the user's emotions and instructions, and makes specific customization suggestions to the user. The input is emotion data and user instructions, and the output is customization suggestions.

[1584] Step 7:

[1585] The user provides customization instructions through a dialogue interface. The generative AI model analyzes the instructions and applies modifications to the initially processed photo. The input is the user's customization instructions, and the output is customized photo data.

[1586] Step 8:

[1587] The emotion engine analyzes the user's emotions in real time and provides feedback to the generative AI. The generative AI model then suggests additional modifications based on the emotions, resulting in optimal customization. The input is emotion data and user feedback, and the output is further modified photo data.

[1588] Step 9:

[1589] The server generates the final edited photo data and creates a permanent link. The server saves the final photo data and provides the user with a download link via email or in-app notification. The input is the final customized photo data, and the output is the download link.

[1590] Step 10:

[1591] The user clicks the provided link and downloads the final photo data through a dedicated app or browser. The user saves the downloaded photos to their device and prints them out or shares them with family and friends as needed. The input is the download link and the output is the downloaded photo data.

[1592] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1593] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1594] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1595] [Fourth embodiment]

[1596] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1597] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1598] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1599] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1600] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1601] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1602] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1603] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1604] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1605] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1606] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1607] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1608] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1609] The following describes in detail an embodiment of the present invention. The present invention is a system for converting old black-and-white photographs to color and sharpening out-of-focus photographs. This system is implemented through a dedicated app that is easy for users to operate, and includes the following processing flow:

[1610] Taking and uploading photos

[1611] 1. The user prepares an old black-and-white photo and launches the dedicated app.

[1612] 2. The device (smartphone or tablet) starts the camera and the user takes a photo. A shooting guide is displayed, instructing the user on the appropriate shooting angle and lighting conditions.

[1613] 3. The user can preview the photo they have taken and retake it if necessary.

[1614] 4. The user selects the photo they are satisfied with and presses the "Upload" button in the dedicated app.

[1615] 5. The device compresses the selected photo data and sends it to the server.

[1616] AI-based initial processing

[1617] 1. The server receives the photo data and runs the image correction AI. First, it converts the monochrome photo to color. This process involves an algorithm that colors each pixel to create natural-looking tones.

[1618] 2. A focus correction algorithm is then used to detect blurry areas in the colorized photo and a sharpening filter is applied, resulting in a clearer image.

[1619] Customization Process

[1620] 1. The server uses the generative AI model to generate a dialogue interface with the user, through which the user can provide specific customization instructions.

[1621] 2. Users can interact with the AI ​​to adjust the colors to their liking or remove unnecessary background elements. For example, they can communicate specific requests such as "I want the background blurred" or "I want the costume color changed to blue."

[1622] 3. The generative AI model analyzes the user's instructions and applies real-time corrections to the initially processed photo.

[1623] 4. The server displays a preview of the edited result to the user and asks for confirmation. The user can make further adjustments while viewing the preview.

[1624] Final data storage and distribution

[1625] 1. The server generates the final edited photo data and creates a permanent link.

[1626] 2. The server saves the final photo data and provides the user with a download link via email or in-app notification.

[1627] 3. The user clicks on the provided link and downloads the final photo data via a dedicated app or browser.

[1628] 4. Users can save the downloaded photos to their own devices and print them out or share them with family and friends as needed.

[1629] Specific examples

[1630] Example 1: Colorizing a black and white wedding photo

[1631] 1. The user launches the app and takes an old black-and-white wedding photo with their camera.

[1632] 2. The device sends the captured image data to the server.

[1633] 3. The server colorizes and sharpens the received photo.

[1634] 4. The server interacts with the user through the generated AI model and receives instructions on adjusting the background.

[1635] 5. The server generates the final retouched photo and provides the user with a download link.

[1636] 6. User downloads and saves the final photo via the link.

[1637] Example 2: Sharpening an out-of-focus photo of grandparents

[1638] 1. The user digitizes old photos using a scanner and uploads them using a dedicated app.

[1639] 2. The device sends the scanned data to the server.

[1640] 3. The server receives the data and performs focus correction and colorization processing.

[1641] 4. The server interacts with the generated AI model and modifies facial details based on the user's wishes.

[1642] 5. The server generates the final photo after user confirmation and provides a download link.

[1643] 6. Users download, save and share.

[1644] By implementing this invention, old black-and-white photographs can be easily colorized and out-of-focus photographs can be sharpened, allowing users to vividly relive past memories. This makes the system extremely useful, especially for people in their 50s and older.

[1645] The processing flow will be explained below.

[1646] Step 1:

[1647] The user prepares an old black-and-white photo and launches the dedicated app.

[1648] Step 2:

[1649] The device (smartphone or tablet) starts the camera and the user takes a photo. A shooting guide is displayed, instructing the appropriate shooting angle and lighting conditions.

[1650] Step 3:

[1651] The user can preview the photo they have taken and take another photo if necessary.

[1652] Step 4:

[1653] The user selects the photo they are satisfied with and presses the "Upload" button in the dedicated app.

[1654] Step 5:

[1655] The terminal compresses the selected photo data and transmits it to the server.

[1656] Step 6:

[1657] The server receives the photo data and temporarily stores it in an input database, where it checks the data for consistency and for missing or error information.

[1658] Step 7:

[1659] The server runs an image correction AI to convert the received monochrome photo into color, which then colors each pixel to produce a natural color tone.

[1660] Step 8:

[1661] The server passes the colorized photo to a focus correction algorithm, which detects blurry areas and applies a sharpening filter.

[1662] Step 9:

[1663] The server sends the color photo data after initial processing to the generative AI model.

[1664] Step 10:

[1665] The server generates an interactive interface with the user through the generated AI model.

[1666] Step 11:

[1667] The user can use the dialogue interface to make color adjustments or remove unnecessary backgrounds, for example, by inputting instructions such as "I want the background to be blurred" or "I want the color of the costumes to be changed to blue."

[1668] Step 12:

[1669] The generative AI model analyzes the user's instructions and applies corrections to the initially processed photo.

[1670] Step 13:

[1671] The server displays the correction results in real time and asks the user for confirmation, allowing the user to give further instructions while viewing the preview and adjust the final result.

[1672] Step 14:

[1673] The server generates the final edited photo data and creates a permanent link.

[1674] Step 15:

[1675] The server stores the final photo data and provides the user with a download link via email or in-app notification.

[1676] Step 16:

[1677] The user clicks on the provided link and downloads the final photo data via a dedicated app or browser.

[1678] Step 17:

[1679] Users can save the downloaded photos to their own devices and print them out or share them with family and friends as needed.

[1680] Example 1

[1681] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1682] Conventional photo correction systems often require manual work, whether it's converting old black-and-white photos to color or correcting out-of-focus photos to make them clearer, making the process extremely time-consuming. Furthermore, specific corrections and customizations require users to have advanced skills, making them difficult for average users without specialized knowledge. For this reason, there is a demand for a system that allows users to easily perform high-quality photo corrections.

[1683] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1684] In this invention, the server includes a device that takes or scans old black-and-white photographs, a device that transmits the taken or scanned black-and-white photograph data to the server, a program that performs image correction on the server and converts the black-and-white photographs to color, a program that corrects the focus of the colorized photographs to make them clearer, a program that interacts with the user using a generative AI model and customizes the photographs based on the user's instructions, and a device that finally provides the customized photographs to the user. This allows users to easily perform high-quality photo correction without specialized knowledge.

[1685] An "old black and white photograph" is a black and white (monochrome) photograph taken in the past, and refers to an image that does not contain color information.

[1686] A "photographing or scanning device" is a device for converting a physical photograph into a digital image, and includes hardware devices such as a camera or scanner.

[1687] "Transmitting device" refers to a hardware or software device for transferring photographed or scanned digital image data to a server via a network.

[1688] "Server" refers to a remote computer or cloud-based computing system for receiving and processing digital image data.

[1689] "Image correction program" refers to a software application that performs operations such as adjusting color tone and correcting exposure on digital images.

[1690] A "color conversion program" refers to a software algorithm that adds color information to each pixel of a monochrome image to convert it into a natural color image.

[1691] A "focus and sharpening program" refers to software that detects blurry areas in an image and applies a sharpening filter to improve the image.

[1692] "Generative AI model" refers to an artificial intelligence model that interacts with users, analyzes their instructions based on natural language prompts, and is responsible for making corrections and improvements to images.

[1693] "Customization program" refers to software that modifies images based on a user's specific needs.

[1694] The "providing device" refers to a device for distributing the final edited image data to the user, and includes email, cloud storage services, etc.

[1695] The system of this invention is mainly designed to convert old black and white photos into color and sharpen out-of-focus photos, allowing users to vividly relive their past memories. The whole system consists of three main components: the user, the terminal, and the server.

[1696] First, the user prepares an old black-and-white photograph and launches the dedicated application. This application runs on devices such as smartphones and tablets, and activates the camera to take a photo. Shooting guides are displayed, allowing the user to take a photo with the optimal lighting and angle.

[1697] The device then displays a preview of the captured photo, allowing the user to review and retake the photo until satisfied. Once a satisfactory photo is selected, the device compresses the photo data and sends it to the server.

[1698] The server receives the photo data and performs image correction. The image correction process first converts the monochrome photo to color. Specifically, it uses Python's OpenCV library to apply an algorithm that estimates the appropriate color for each pixel and generates natural-looking tones. Next, a focus correction algorithm using TensorFlow is applied, detecting blurred areas and applying a sharpening filter to make the entire photo clearer.

[1699] In addition, a customization process is provided in which users interact with a generative AI model. This generative AI model uses advanced natural language processing techniques, such as GPT-4, to modify photos based on prompts entered by the user. Users can provide specific customization instructions through the dialogue interface, such as "blur the background" or "change the color of the outfit to blue." The generative AI model analyzes these instructions and applies them to the photo in real time.

[1700] The final customized photo is generated as high-resolution image data by the server. The server stores this data in cloud storage and generates a permanent link that is provided to the user. The user receives a download link via email or in-app notification, and can download the final photo data through a dedicated application or browser and save it to their device. This allows the user to print out the final photo or share it with family and friends.

[1701] As a concrete example, we will show the process of colorizing a black-and-white wedding photo. The user launches a dedicated app and takes a photo of an old black-and-white wedding photo with their camera. The device sends the captured data to a server, which then colorizes and sharpens the received photo. The background is then adjusted based on the user's instructions via a generative AI model, and the user is provided with a download link for the final edited photo.

[1702] By implementing this invention, users can easily colorize old monochrome photographs and sharpen out-of-focus photographs without any specialized knowledge. This provides a very useful system for vividly reviving past memories, especially for those in their 50s and older.

[1703] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1704] Step 1:

[1705] The user prepares an old black-and-white photo and launches the dedicated app.

[1706] Input: Physical black and white photo

[1707] Output: Launching the dedicated app

[1708] Specific operation: The user prepares a physical black-and-white photo and launches a dedicated app on their smartphone or tablet.

[1709] Step 2:

[1710] The device will activate the camera and take a photo.

[1711] Input: User-initiated camera activation and shooting instructions

[1712] Output: Digitized black and white photo capture

[1713] Specific operation: The device's camera is activated, a shooting guide is displayed, and the user presses the shooting button to digitize a monochrome photo.

[1714] Step 3:

[1715] The device displays a preview of the photo for the user to review and retake.

[1716] Input: Digitized black and white photograph

[1717] Output: Show preview image

[1718] Specific operation: The device displays a preview of the photo taken and displays the "Confirm" and "Retake" buttons. The user checks the preview and presses the "Retake" button if they need to take another photo.

[1719] Step 4:

[1720] The user selects a photo they are satisfied with and presses the upload button.

[1721] Input: A preview image selected by the user.

[1722] Output: Ready to upload status

[1723] Specific operation: The user presses the "Confirm" button, which takes them to the upload screen of the dedicated app. The user presses the "Upload" button.

[1724] Step 5:

[1725] The terminal compresses the selected photo data and transmits it to the server.

[1726] Input: A digitized black and white photograph selected by the user

[1727] Output: Compressed photo data sent to the server

[1728] Specific operation: The device compresses the image data and sends it to a server via the Internet.

[1729] Step 6:

[1730] The server receives the photo data and performs image correction.

[1731] Input: Compressed photo data

[1732] Output: Colorized and focused image

[1733] What it does: The server receives the photo data and colorizes the monochrome photo using Python's OpenCV library, then uses TensorFlow to apply a focus correction algorithm to make the photo clearer.

[1734] Step 7:

[1735] The server uses the generative AI model to generate a dialogue interface with the user.

[1736] Input: Colorized and focus-corrected image data

[1737] Output: Generate a conversational interface

[1738] Specific operation: The server uses a generative AI model (e.g., GPT-4) to generate a dialogue interface and provides a screen for the user to input customization instructions.

[1739] Step 8:

[1740] The user gives specific customization instructions.

[1741] Input: Prompt text given by the user (e.g., "I want the background to be blurred" or "I want the color of the costume to be blue")

[1742] Output: Instruction analysis results by the generative AI model

[1743] Specific operation: The user inputs a prompt sentence through the dialogue interface, and the AI ​​receives and analyzes it.

[1744] Step 9:

[1745] A generative AI model modifies photos based on user instructions.

[1746] Input: Prompt statement analysis result

[1747] Output: Corrected image data

[1748] How it works: The generative AI model modifies specific parts of the image based on user instructions and makes corresponding adjustments in real time.

[1749] Step 10:

[1750] The server displays a preview of the correction results to the user.

[1751] Input: Corrected image data

[1752] Output: Preview screen display

[1753] What happens: The server previews the corrected image to the user, allowing the user to make further adjustments.

[1754] Step 11:

[1755] The server generates the final edited photo data and creates a fixed link.

[1756] Input: Final corrected image data

[1757] Output: Cloud storage of image data and generation of a permanent link

[1758] Specific operation: The server generates a high-resolution image after correction is completed, saves it to cloud storage, and then creates a permanent link.

[1759] Step 12:

[1760] The server notifies the link and provides it to the user.

[1761] Input:permalink

[1762] Output: User notification (email or in-app notification)

[1763] Specific operation: The server notifies the user of the fixed link and provides a download link.

[1764] Step 13:

[1765] The user clicks on the provided link to download the final photo data.

[1766] Input:permalink

[1767] Output: Final downloaded photo data

[1768] Specific operation: The user clicks on the link and downloads the final photo data via a dedicated app or web browser.

[1769] Step 14:

[1770] Save the photos you download to your device and share them as needed.

[1771] Input: Downloaded photo data

[1772] Output: Photos stored on the device and shared photos

[1773] What it does: Users can save the downloaded photos to their devices, share them with family and friends, and even print them out.

[1774] (Application example 1)

[1775] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1776] In recent years, advances in digital technology have led to an increasing number of users wanting to digitize and preserve old photographs. However, converting old black-and-white photographs to color and sharpening blurry photographs is a time-consuming process that requires specialized knowledge. Furthermore, there are limited means to easily provide such services, especially in brick-and-mortar stores. This makes it difficult for customers to restore photos at their own convenience, leading to lower service satisfaction. Therefore, there is a need for a system that can easily colorize and sharpen old photographs in-store.

[1777] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1778] In this invention, the server includes means for taking or scanning old monochrome photographs, means for transmitting the taken or scanned monochrome photograph data to the server, means for performing image correction on the server to convert the monochrome photograph to color, means for correcting the focus of the colorized photograph to make it clearer, means for interacting with the user using a generative AI and customizing the photograph based on the user's instructions, means for providing the user with the final customized photograph, and means for the user to use the application that performs the photo customization process in a physical store environment. This allows customers to easily colorize and sharpen old photographs in a physical store.

[1779] An "old black-and-white photograph" is a photograph taken in the past that lacks color and is expressed in black and white rather than color.

[1780] "Means of photographing or scanning" refers to the use of a device such as a camera or scanner to obtain a photograph as digital data.

[1781] A "transmitting means" is a communication means for sending digital data over the Internet or a local network to a remote server.

[1782] A "server" is a computer system that receives and processes digital data.

[1783] "Image correction means" refers to software or algorithms for adjusting contrast and correcting color of digital images.

[1784] "Methods for converting to color" are techniques for adding color to monochrome photographs to reproduce them as natural color photographs.

[1785] "Focus correction and sharpening" is a method for sharpening blurry photos and improving the clarity of details.

[1786] "Generative AI" is a system that uses an artificial intelligence model to interact with users in natural language and process images based on their explanations.

[1787] "Means for interacting with users and customizing" refers to technology that uses AI to understand users' requests and modify or process specific parts of images.

[1788] A "means for providing a customized photograph to a user" is a method by which the final, enhanced digital image can be downloaded or viewed by the user.

[1789] "Methods available to users in a physical store environment" refers to the use of applications and digital devices by customers to receive services within a physical store.

[1790] The following describes in detail the mode for carrying out the invention. This system has various functions for converting old black-and-white photos to color and sharpening out-of-focus photos. It is intended for use in brick-and-mortar stores, and customers can easily use these functions using a dedicated application.

[1791] Hardware and software used

[1792] The system uses the following hardware and software:

[1793] Hardware:

[1794] Devices (smartphones, tablets, cameras)

[1795] Server (high performance processor, internet connection)

[1796] software:

[1797] OpenCV (image capture, image processing)

[1798] PIL (Python Imaging Library) (Image Processing)

[1799] requests (HTTP communication library)

[1800] Generative AI model (user interaction, image customization)

[1801] Processing flow explanation

[1802] Taking and uploading photos

[1803] The user launches a dedicated app and prepares an old black-and-white photo. The device then activates the camera, and the user takes a photo. A shooting guide is displayed, instructing on the appropriate shooting angle and lighting conditions. The user can check a preview of the photo they have taken and take another photo if necessary. When the user selects a photo they are satisfied with and presses the upload button, the device compresses the photo data and sends it to the server.

[1804] AI-based initial processing

[1805] The server receives the photo data and runs an image correction AI, first converting the black and white photo to color, then using a focus correction algorithm to detect blurry areas in the colorized photo and applying a sharpening filter.

[1806] Customization Process

[1807] The server uses the generative AI model to generate a dialogue interface for the user. The user can use this interface to give specific customization instructions. For example, they can express specific requests such as "I want the background blurred" or "I want the costume color changed to blue." The generative AI model analyzes the user's instructions and applies real-time corrections to the initially processed photo. The server then displays a preview of the corrections to the user and receives confirmation and additional instructions.

[1808] Final data storage and distribution

[1809] The server generates the final edited photo data and creates a permanent link. The server saves the final photo data and provides the user with a download link via email or in-app notification. The user clicks the link and downloads the final photo data via a dedicated app or browser. The user saves the downloaded photo to their device and can print it out or share it with family and friends as needed.

[1810] Examples of concrete examples and prompts

[1811] As a concrete example, customers can take old family photos with the app in a physical store where their families gather, and have them colorized and sharpened on the spot. This service allows customers to vividly relive their past memories and share them as new memories.

[1812] Example prompt sentence:

[1813] 1. Take a photo or upload an existing one.

[1814] 2. Sending photos to server... Please wait.

[1815] 3. Processing the photo… We will correct the focus and add color to the original photo.

[1816] 4. Processing completed! Please check.

[1817] 5. Do you want to customize further? For example, "I want the background to be blurred" or "I want the costume color to be changed to blue."

[1818] This system allows customers to easily colorize and sharpen old photos in-store.

[1819] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1820] Step 1:

[1821] Photographing and scanning

[1822] The user prepares an old black-and-white photo and launches a dedicated app. The device launches the camera and the user takes a photo. A shooting guide is displayed, instructing on the appropriate shooting angle and lighting conditions. The input is the old black-and-white photo, and the output is a digital copy of the photo. A preview is provided to confirm that the appropriate shooting conditions are met, allowing the user to decide whether or not to retake the photo.

[1823] Step 2:

[1824] Upload

[1825] When the user selects a photo they are satisfied with and presses the upload button, the device compresses the photo data and sends it to the server. The input is a digital photo, and the output is compressed photo data. This process improves data transfer speed and reduces server load.

[1826] Step 3:

[1827] Initial Data Processing

[1828] The server receives the photo data and first runs the image correction AI. The input is compressed photo data, and the output is colorized photo data. Specifically, it applies an algorithm that adds color to each pixel of the monochrome photo and generates natural-looking tones.

[1829] Step 4:

[1830] Focus Correction

[1831] The server then uses a focus correction algorithm to detect blurred areas in the colorized photo and applies a sharpening filter. The input is the colorized photo data, and the output is the sharpened photo data. Specifically, the server runs a filter to improve the contrast and detail in the blurred areas.

[1832] Step 5:

[1833] Customization with generative AI

[1834] The server uses the generative AI model to generate a dialogue interface for the user. The user can then provide specific customization instructions through this interface. The input is the user's text-based instructions, and the output is customized photo data. Specific operations include analyzing requests such as "blur the background" or "change the color of the costume to blue," and reflecting the results in image processing.

[1835] Step 6:

[1836] Preview and fix

[1837] The server displays a preview of the correction results to the user and receives confirmation and additional instructions from the user. The input is customized photo data, and the output is the final corrected photo data. Specifically, it provides an interface that allows the user to make further adjustments while viewing the preview.

[1838] Step 7:

[1839] Final data storage and distribution

[1840] The server generates the final corrected photo data and creates a permanent link. The server saves the final photo data and provides the user with a download link via email or in-app notification. The input is the final corrected photo data and the output is a permanent link. Specifically, this involves saving the photo in a database and generating a link to provide it to the user.

[1841] Through this process, customers can easily colorize and sharpen old photos in-store.

[1842] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1843] The following describes in detail the embodiments of the present invention. The present invention combines a system for colorizing old black-and-white photos and sharpening out-of-focus photos with an emotion engine that recognizes the user's emotions. This system is implemented through a dedicated app that is easy for users to operate, and includes the following processing flow:

[1844] Taking and uploading photos

[1845] 1. The user prepares an old black-and-white photo and launches the dedicated app.

[1846] 2. The device (smartphone or tablet) starts the camera and the user takes a photo. A shooting guide is displayed, instructing the user on the appropriate shooting angle and lighting conditions.

[1847] 3. The user can preview the photo they have taken and retake it if necessary.

[1848] 4. The user selects the photo they are satisfied with and presses the "Upload" button in the dedicated app.

[1849] 5. The device compresses the selected photo data and sends it to the server.

[1850] AI-based initial processing

[1851] 1. The server receives the photo data and temporarily stores it in an input database, where it checks the data for consistency and omissions or errors.

[1852] 2. The server runs the image correction AI to convert the received monochrome photo to color. The image correction AI colors each pixel to produce natural-looking colors.

[1853] 3. The server passes the colorized photo to a focus correction algorithm, which detects blurry areas and applies a sharpening filter.

[1854] Emotion engine processing

[1855] 1. The server activates the emotion engine and uses the camera and microphone to analyze the user's voice and facial expressions.

[1856] 2. The device transmits the user's voice and facial expression data to the emotion engine in real time.

[1857] 3. The emotion engine analyzes the received data and determines the user's emotion. For example, if the user is smiling, it is determined to be "joy," and if they are frowning, it is determined to be "confusion."

[1858] 4. The server receives the judgment results from the emotion engine and works with the generation AI.

[1859] Customization Process

[1860] 1. The server uses the generative AI model to generate a dialogue interface with the user.

[1861] 2. The user enters specific customization instructions through the dialogue interface, such as "I want the background to be blurred" or "I want the color of the costume to be blue."

[1862] 3. The generative AI model analyzes the user's instructions and applies corrections to the initially processed photo.

[1863] 4. The emotion engine analyzes the user's emotions in real time and provides feedback to the generative AI, allowing the generative AI model to suggest additional modifications based on the emotions.

[1864] 5. The server displays the edited results in real time and asks the user for confirmation. The user can make further adjustments while viewing the preview.

[1865] Final data storage and distribution

[1866] 1. The server generates the final edited photo data and creates a permanent link.

[1867] 2. The server saves the final photo data and provides the user with a download link via email or in-app notification.

[1868] 3. The user clicks on the provided link and downloads the final photo data via a dedicated app or browser.

[1869] 4. Users can save the downloaded photos to their own devices and print them out or share them with family and friends as needed.

[1870] Specific examples

[1871] Example 1: Colorization of Black and White Wedding Photos and Emotion-Based Customization

[1872] 1. The user launches the app and takes an old black-and-white wedding photo with their camera.

[1873] 2. The device sends the captured image data to the server.

[1874] 3. The server colorizes and sharpens the received photo.

[1875] 4. The server interacts with the user through the generative AI model and emotion engine and receives instructions on adjusting the background.

[1876] 5. The emotion engine detects the user's happiness and suggests warmer background colors.

[1877] 6. The server generates the final retouched photo and provides the user with a download link.

[1878] 7. User downloads and saves the final photo via the link.

[1879] Example 2: Sharpening out-of-focus photos of grandparents and emotion-based customization

[1880] 1. The user digitizes old photos using a scanner and uploads them using a dedicated app.

[1881] 2. The device sends the scanned data to the server.

[1882] 3. The server receives the data and performs focus correction and colorization processing.

[1883] 4. The server interacts with the user through the generative AI model and emotion engine, and performs facial detail correction.

[1884] 5. The emotion engine analyzes the user's emotions, and if it detects "confusion," the generative AI will suggest more detailed adjustment methods.

[1885] 6. The server generates the final photo after user confirmation and provides a download link.

[1886] 7. Users can download, save, or share.

[1887] By implementing this invention, not only can old black-and-white photographs be easily colorized and out-of-focus photographs be made clearer, but the system can also be optimally customized based on the user's emotions, allowing users to relive their past memories in a richer way. This makes the system extremely useful, especially for those in their 50s and older.

[1888] The processing flow will be explained below.

[1889] Step 1:

[1890] The user prepares an old black-and-white photo and launches the dedicated app.

[1891] Step 2:

[1892] The device (smartphone or tablet) starts the camera and the user takes a photo. A shooting guide is displayed, instructing the appropriate shooting angle and lighting conditions.

[1893] Step 3:

[1894] The user can preview the photo they have taken and take another photo if necessary.

[1895] Step 4:

[1896] The user selects the photo they are satisfied with and presses the "Upload" button in the dedicated app.

[1897] Step 5:

[1898] The terminal compresses the selected photo data and transmits it to the server.

[1899] Step 6:

[1900] The server receives the photo data and temporarily stores it in an input database, where it checks the data for consistency and for missing or error information.

[1901] Step 7:

[1902] The server runs an image correction AI to convert the received monochrome photo into color, which then colors each pixel to produce a natural color tone.

[1903] Step 8:

[1904] The server passes the colorized photo to a focus correction algorithm, which detects blurry areas and applies a sharpening filter.

[1905] Step 9:

[1906] The server runs the emotion engine and uses the camera and microphone to analyze the user's voice and facial expressions.

[1907] Step 10:

[1908] The device transmits the user's voice and facial expression data to the emotion engine in real time.

[1909] Step 11:

[1910] The emotion engine analyzes the received data and determines the user's emotion. For example, if the user is smiling, it is determined to be "joy," and if they are frowning, it is determined to be "confusion."

[1911] Step 12:

[1912] The server receives the judgment results from the emotion engine and works in conjunction with the generation AI.

[1913] Step 13:

[1914] The server uses the generative AI model to generate a dialogue interface with the user.

[1915] Step 14:

[1916] The user enters specific customization instructions through a dialogue interface, such as "I want the background to be blurred" or "I want the color of the costume to be changed to blue."

[1917] Step 15:

[1918] The generative AI model analyzes the user's instructions and applies corrections to the initially processed photo.

[1919] Step 16:

[1920] The emotion engine analyzes the user's emotions in real time and provides feedback to the generative AI.

[1921] Step 17:

[1922] The generative AI model suggests additional modifications based on the user's sentiment.

[1923] Step 18:

[1924] The server displays the correction results in real time and asks the user for confirmation, allowing the user to make further adjustments while viewing the preview.

[1925] Step 19:

[1926] The server generates the final edited photo data and creates a permanent link.

[1927] Step 20:

[1928] The server stores the final photo data and provides the user with a download link via email or in-app notification.

[1929] Step 21:

[1930] The user clicks on the provided link and downloads the final photo data via a dedicated app or browser.

[1931] Step 22:

[1932] Users can save the downloaded photos to their own devices and print them out or share them with family and friends as needed.

[1933] Example 2

[1934] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1935] Conventional photo restoration systems provide methods for colorizing old monochrome photographs and correcting focus, but do not offer customization that takes into account the user's emotions. As a result, it takes a lot of time and effort to achieve results that satisfy the user. The purpose of this invention is to solve these problems and provide a system that is easy for users to operate and that allows customization based on emotions.

[1936] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1937] In this invention, the server includes means for taking or scanning old monochrome photographs, means for transmitting the taken or scanned monochrome photograph data to an information processing device, means for performing image correction on the information processing device to convert the monochrome photograph to color, means for correcting the focus of the colorized photograph to make it clearer, means for analyzing the user's voice and facial expression in real time to determine their emotions, means for interacting with the user using a generative AI and customizing the photo based on the user's instructions and the determined emotions, and means for finally providing the customized photo to the user. This enables photo restoration and customization that satisfies the user to be performed quickly and easily.

[1938] "Old monochrome photographs" are black and white photographic data that were taken in the past and do not contain color information.

[1939] "Photographing or scanning means" refers to methods and devices that use cameras or scanners to convert physical photographs or images into digital data.

[1940] "Information processing device" refers to a computer system such as a server or cloud computing system for processing captured or scanned image data.

[1941] "Method of image correction and converting black-and-white photos to color" refers to an image processing technology that assigns an appropriate color to each pixel of a black-and-white photo to reproduce natural color tones.

[1942] "Method for correcting focus and sharpening colorized photographs" refers to a technique that detects blurry areas of an image and applies a sharpening filter to them to increase the overall clarity of the image.

[1943] "Means for analyzing a user's voice and facial expressions in real time to determine emotions" refers to a system that uses a microphone or camera to collect a user's voice and facial expressions, and analyzes that data to identify the user's emotional state.

[1944] "Generative AI" refers to an artificial intelligence system that interacts with users based on input data and instructions and automatically performs the necessary customizations.

[1945] "Means for interacting with a user and customizing a photo based on the user's instructions and determined emotions" refers to a method and apparatus that receives instructions from a user via a user interface and further processes or modifies a photo taking into account the user's emotional state.

[1946] "Means for providing the final customized photo to the user" means a system that provides the final processed photo data to the user via a download link or in-app notification.

[1947] This system aims to restore a clear image by colorizing old monochrome photographs and adjusting the focus. It also analyzes the user's voice and facial expressions to determine their emotions, and customizes the photo restoration process based on those emotions to ensure the user is satisfied with the results. This system is implemented using the following hardware and software:

[1948] Hardware configuration:

[1949] Device: A device such as a smartphone, tablet, or computer that contains a camera and microphone and collects user input and emotional data.

[1950] Server: A server with high-performance processing power, where image processing, database management, and AI model execution take place.

[1951] Software configuration:

[1952] Image Enhancement Software: This includes AI algorithms that colorize old black-and-white photos and correct focus. Specific examples of the software used include OpenCV and TensorFlow.

[1953] Emotion engine: Software that analyzes the user's voice and facial expressions to determine their emotions. Specific software used includes "DeepFace" and "Google Cloud Speech-to-Text."

[1954] Generative AI models: Personalize photos based on user input and emotions, using natural language processing and generative models like GPT-3 and DALL·E.

[1955] Examples:

[1956] 1. Colorization of monochrome wedding photos and emotion-based customization

[1957] The user launches the dedicated app and takes an old black-and-white wedding photo with their camera.

[1958] The device sends the captured image data to the server.

[1959] The server then colorizes and sharpens the received photographs.

[1960] The server interacts with the user through the generative AI model and emotion engine, and receives instructions for adjusting the background.

[1961] The emotion engine detects the user's happiness and suggests warmer background colors.

[1962] The server generates the final retouched photo and provides the user with a download link.

[1963] The user downloads and saves the final photo via a link.

[1964] 2. Sharpening out-of-focus photos of grandparents and emotion-based customization

[1965] Users digitize old photos using a scanner and upload them using a dedicated app.

[1966] The device sends the scanned data to the server.

[1967] The server receives the data and performs focus correction and colorization processing.

[1968] The server interacts with the user through the generative AI model and emotion engine, and performs facial detail correction.

[1969] The emotion engine analyzes the user's emotions, and if it detects "confusion," the generative AI will suggest more detailed adjustment methods.

[1970] The server generates the final photo after user confirmation and provides a download link.

[1971] Users can download, save and share.

[1972] Example prompt sentence:

[1973] 1. "Blur the background in this photo to make the person stand out."

[1974] 2. "Please change the color of the outfit in this photo to blue to balance the overall look."

[1975] 3. "I'd like to see a bit more sharpness in this photo to bring out the details."

[1976] This system allows users to easily colorize old black-and-white photos, improve their clarity, and even customize them individually, allowing users to more vividly relive their rich memories.

[1977] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1978] Step 1:

[1979] The user prepares an old black-and-white photo and launches the dedicated app.

[1980] Input: Old black and white photo

[1981] Output: Ready to be photographed or scanned

[1982] Step 2:

[1983] When the user presses the "Take Photo" button on the dedicated app, the device's camera will launch and a photo guide will be displayed.

[1984] Input: User action (pressing the "Take Photo" button)

[1985] Output: Starts the camera and displays the shooting guide

[1986] What it does: The shooting guide displays messages instructing you on the appropriate shooting angles and lighting conditions.

[1987] Step 3:

[1988] The user takes a photo using the camera and then sees a preview screen after taking the photo, which allows them to choose whether to retake the photo or use the one they just took.

[1989] Input: Photo data taken

[1990] Output: Preview screen and user choice (retake or use)

[1991] Step 4:

[1992] The user selects a photo they are satisfied with and presses the "Upload" button in the dedicated app.

[1993] Input: User operation (pressing the "Upload" button)

[1994] Output: Photo ready to upload

[1995] Step 5:

[1996] The device compresses the selected photo data and sends it to the server. The device uses a compression algorithm to minimize the data size and reduce transmission time.

[1997] Input: Photo data taken

[1998] Output: Compressed photo data, sent to server

[1999] Step 6:

[2000] The server receives the photo data and temporarily stores it in an input database, where it checks the data for consistency and omissions or errors.

[2001] Input: Compressed photo data

[2002] Output: Data stored in the database, consistency check results

[2003] Step 7:

[2004] The server runs an image correction AI to convert the received monochrome photo into color, assigning the appropriate color to each pixel and reproducing natural color tones.

[2005] Input: Monochrome photo data

[2006] Output: Colorized photo data

[2007] How it works: An AI algorithm colors each image pixel by pixel, ensuring a natural overall color tone.

[2008] Step 8:

[2009] The server passes the colorized photo through a focus correction algorithm, which detects blurry areas and applies a sharpening filter.

[2010] Input: Colorized photo data

[2011] Output: Focused photo data

[2012] What it does: Image analysis identifies blurry areas and applies a sharpening filter.

[2013] Step 9:

[2014] The server starts the emotion engine and initializes the camera and microphone to analyze the user's voice and facial expressions.

[2015] Input: User voice and facial expression data

[2016] Output: Initial setup completed

[2017] What it does: Camera and microphone settings are automatically adjusted, and voice and facial expression data is captured in real time.

[2018] Step 10:

[2019] The device transmits the user's voice and facial expression data to the emotion engine in real time.

[2020] Input: User's voice and facial expression data

[2021] Output: Data sent to the emotion engine

[2022] Step 11:

[2023] The emotion engine analyzes the received data and determines the user's emotion.

[2024] Input: Voice and facial expression data

[2025] Output: Sentiment analysis result (e.g., happy, confused)

[2026] Specific operation: The AI ​​algorithm analyzes voice tone, facial features, etc. to determine emotions.

[2027] Step 12:

[2028] The server passes the judgment results from the emotion engine to the generation AI and works in conjunction with the generation AI.

[2029] Input: Sentiment analysis results

[2030] Output: Data passed to the generation AI

[2031] Specific operation: Emotional data is provided to the generative AI, which generates customizations based on the user's emotions.

[2032] Step 13:

[2033] The server uses the generated AI model to generate a dialogue interface with the user.

[2034] Input: Generative AI model held by the server

[2035] Output: Generates a conversational interface

[2036] Specific operation: Using natural language processing, it dynamically generates an easy-to-use dialogue interface for users.

[2037] Step 14:

[2038] The user inputs specific customization instructions through the dialogue interface.

[2039] Input: User customization instructions

[2040] Output: Instruction data sent to the server

[2041] Step 15:

[2042] The generative AI model analyzes the user's instructions and applies corrections to the initially processed photo.

[2043] Input: User customization instructions and initial processed photo data

[2044] Output: Photo data with corrections

[2045] What it does: Based on user instructions, the image editing algorithm applies specific modifications.

[2046] Step 16:

[2047] The emotion engine analyzes the user's emotions in real time and provides feedback to the generative AI.

[2048] Input: Real-time user emotion data

[2049] Output: Feedback provided to the generative AI

[2050] What it does: The updated emotion data is passed to the generative AI, providing reference information for further customization.

[2051] Step 17:

[2052] The server displays the results of the modifications in real time and asks the user for confirmation, allowing the user to make further adjustments while viewing the preview.

[2053] Input: Modified photo data and user emotion data

[2054] Output: Preview screen and user adjustment instructions

[2055] Step 18:

[2056] The server generates the final edited photo data and creates a permanent link.

[2057] Input: Final edited photo data

[2058] Output: Download link

[2059] Step 19:

[2060] The server stores the final photo data and provides the user with a download link via email or in-app notification.

[2061] Input: Last modified photo data and user contact information

[2062] Output: Provide a download link

[2063] Step 20:

[2064] The user clicks on the provided link and downloads the final photo data via a dedicated app or browser.

[2065] Input: Download link

[2066] Output: Downloaded photo data

[2067] Step 21:

[2068] Users can save the downloaded photos to their own devices and print them out or share them with family and friends as needed.

[2069] Input: Downloaded photo data

[2070] Output: Saved and shared photos

[2071] (Application example 2)

[2072] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2073] Conventional photo editing systems simply colorize and correct uniform photos without considering the user's emotions, making it difficult to achieve the optimal customization desired by the user. Furthermore, while the advertising industry requires the generation of advertising materials based on user emotions, current technology has difficulty providing this efficiently and effectively. Therefore, there is a need for a system that can analyze user emotions and customize photos based on them.

[2074] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2075] In this invention, the server includes means for taking or scanning old monochrome photographs, means for transmitting the taken or scanned monochrome photograph data to the server, means for performing image correction on the server to convert the monochrome photograph to color, means for correcting the focus of the colorized photograph to make it clearer, means for analyzing emotions from the user's voice and facial expressions, means for interacting with the user using a generative AI and customizing the photo based on the user's instructions and emotional analysis, and means for finally providing the customized photo to the user. This enables optimal photo customization and generation of advertising materials that take the user's emotions into consideration.

[2076] An "old black and white photograph" is a black and white photograph taken a long time ago, and does not contain color information.

[2077] "Means for photographing or scanning" refers to a device or system for digitizing photographs using a camera, scanner, etc.

[2078] "Means for transmitting to a server" refers to a function for transferring digital data to a server via the Internet or other communications network.

[2079] "Image enhancement" refers to digital processing to improve the quality of photographs, particularly color conversion and focus correction.

[2080] The "means for converting to color" is a technique for estimating the color of each pixel in a monochrome photograph and converting it into a color photograph with natural color tones.

[2081] "Focus correction and sharpening" is a technique that detects blurry areas in an image and applies a sharpening filter to make the photo clearer.

[2082] "Means for analyzing emotions from the user's voice and facial expressions" refers to software or algorithms that use a microphone or camera to collect data on the user's voice and facial expressions and analyze the emotions.

[2083] "Means for interacting with users using generative AI" refers to a system that utilizes artificial intelligence to interact with users and receive their instructions.

[2084] The "means for photo customization" is a system for photo editing, such as color adjustment and removal of unwanted background, based on user instructions and emotional analysis.

[2085] The "means for providing customized photos to users" is a mechanism for delivering the final processed photo data to users.

[2086] The present invention provides a system that colorizes old monochrome photos, corrects the focus, and makes them clearer, as well as analyzes the user's emotions and customizes the photos based on those emotions. This system is composed of a server and a terminal, and includes the following means:

[2087] First, the user prepares an old monochrome photograph and takes or scans it using a device (smartphone or tablet). At this time, the device displays a shooting guide that instructs the appropriate shooting angle and lighting conditions. The captured photograph is previewed, and if the user is satisfied with it, it is sent to the server.

[2088] The server temporarily stores the received photo data and verifies its integrity. It then runs an image correction algorithm to convert the monochrome photo to color, using image correction technologies such as OpenCV and TensorFlow. The colorized photo is then passed to a focus correction algorithm to sharpen any blurry areas.

[2089] The server then launches an emotion engine, using the camera and microphone to analyze the user's voice and facial expression data in real time. It performs emotion analysis using Google Cloud's Natural Language API. After determining the user's emotion, it uses a generative AI model (e.g., OpenAI's GPT-4) to generate a dialogue interface for the user. Through this interface, the user can provide specific customization instructions, such as "blur the background" or "change the color of the outfit to blue."

[2090] The emotion engine analyzes the user's emotions in real time and provides feedback to the generative AI, which then customizes the app based on the user's emotions. For example, if the user is using a wedding photo and the "joy" emotion is detected, the generative AI will suggest a warm-toned background.

[2091] As a concrete example, when colorizing and customizing wedding photos, if a user requests that the background be "warm toned," the following prompt sentence can be used for the generative AI model:

[2092] Example prompt sentence:

[2093] The user emotion is "joy." The provided photo is an old black-and-white wedding photo that has been colorized and focused. Create an ad using this photo as a base, suggesting a warm-toned background and an elegant tagline that will inspire users.

[2094] The server then applies the instructions and delivers the final, customized photo to the user. The photo data is then saved and a download link is provided to the user via email or in-app notification. The user can then download and use the final photo data via this link.

[2095] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2096] Step 1:

[2097] The user prepares an old black-and-white photograph and turns on the device (smartphone or tablet). The device displays a shooting guide, instructing on the appropriate shooting angle and lighting conditions. The user takes a photo and checks the preview. The user selects a photo they are satisfied with and presses the "upload" button in the dedicated app. At this point, the input is the digital data of the black-and-white photograph, and the output is photo data ready to be sent to the server.

[2098] Step 2:

[2099] The device compresses the selected photo data and sends it to the server. The input is the photo data stored on the device, and the output is the compressed photo data sent to the server. The server verifies the integrity of the received photo data and checks for omissions or errors.

[2100] Step 3:

[2101] The server runs image correction AI to convert received monochrome photos to color. It uses image correction technologies such as OpenCV and TensorFlow to estimate the color of each pixel and generate natural color tones. The input is monochrome photo data, and the output is colorized photo data.

[2102] Step 4:

[2103] The server passes the colorized photo data to a focus correction algorithm, which detects blurry areas in the image and applies a sharpening filter. The input is the colorized photo data, and the output is the sharpened photo data.

[2104] Step 5:

[2105] The server activates the emotion engine and uses the device's camera and microphone to analyze the user's voice and facial expressions. The device sends the user's voice and facial expression data to the emotion engine in real time. The input is the user's voice and facial expression data, and the output is the analyzed emotion data.

[2106] Step 6:

[2107] The server receives the judgment results from the emotion engine and connects with a generative AI model (e.g., OpenAI's GPT-4). The generative AI model generates a dialogue interface based on the user's emotions and instructions, and makes specific customization suggestions to the user. The input is emotion data and user instructions, and the output is customization suggestions.

[2108] Step 7:

[2109] The user provides customization instructions through a dialogue interface. The generative AI model analyzes the instructions and applies modifications to the initially processed photo. The input is the user's customization instructions, and the output is customized photo data.

[2110] Step 8:

[2111] The emotion engine analyzes the user's emotions in real time and provides feedback to the generative AI. The generative AI model then suggests additional modifications based on the emotions, resulting in optimal customization. The input is emotion data and user feedback, and the output is further modified photo data.

[2112] Step 9:

[2113] The server generates the final edited photo data and creates a permanent link. The server saves the final photo data and provides the user with a download link via email or in-app notification. The input is the final customized photo data, and the output is the download link.

[2114] Step 10:

[2115] The user clicks the provided link and downloads the final photo data through a dedicated app or browser. The user saves the downloaded photos to their device and prints them out or shares them with family and friends as needed. The input is the download link and the output is the downloaded photo data.

[2116] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2117] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2118] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2119] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2120] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2121] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2122] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2123] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2124] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2125] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2126] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2127] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2128] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2129] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2130] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2131] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2132] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2133] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2134] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2135] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2136] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2137] The following is further disclosed regarding the above embodiment.

[2138] (Claim 1)

[2139] A means of photographing or scanning old black-and-white photographs;

[2140] A means for transmitting the captured or scanned monochrome photograph data to a server;

[2141] A method to correct images on the server and convert monochrome photos to color,

[2142] A means of correcting the focus of colorized photographs to make them clearer,

[2143] A means to interact with users using generative AI and customize photos based on their instructions;

[2144] A means to provide users with the final customized photo;

[2145] A system including:

[2146] (Claim 2)

[2147] 2. The system according to claim 1, wherein color conversion and focus correction of monochrome photographs are performed simultaneously.

[2148] (Claim 3)

[2149] The system described in claim 1 is characterized in that color adjustment and removal of unnecessary background are performed through interaction between the user and the generating AI.

[2150] "Example 1"

[2151] (Claim 1)

[2152] A device for taking or scanning old black-and-white photographs;

[2153] a device for transmitting captured or scanned monochrome photograph data to a server;

[2154] A program that performs image correction on the server and converts monochrome photos to color,

[2155] A program to correct the focus of colorized photos and make them clearer,

[2156] A program that uses a generative AI model to interact with users and customize photos based on their instructions.

[2157] a device for providing the final customized photograph to the user;

[2158] A system including:

[2159] (Claim 2)

[2160] 2. The system according to claim 1, wherein color conversion and focus correction of monochrome photographs are performed simultaneously.

[2161] (Claim 3)

[2162] The system described in claim 1 is characterized in that color adjustment and removal of unnecessary background are performed through interaction between the user and the generative AI model.

[2163] "Application Example 1"

[2164] (Claim 1)

[2165] A means of photographing or scanning old black-and-white photographs;

[2166] A means for transmitting the captured or scanned monochrome photograph data to a server;

[2167] A method to correct images on the server and convert monochrome photos to color,

[2168] A means of correcting the focus of colorized photographs to make them clearer,

[2169] A means to interact with users using generative AI and customize photos based on their instructions;

[2170] A means to provide users with the final customized photo;

[2171] A means for users to use the photo customization application in a brick-and-mortar store environment;

[2172] A system including:

[2173] (Claim 2)

[2174] 10. The system of claim 1, wherein color conversion and focus correction of monochrome photographs are performed simultaneously.

[2175] (Claim 3)

[2176] The system of claim 1, characterized in that color adjustment and removal of unnecessary background are performed through interaction between the user and the generating AI.

[2177] "Example 2: Combining Emotion Engines"

[2178] (Claim 1)

[2179] A means of photographing or scanning old black-and-white photographs;

[2180] means for transmitting photographed or scanned monochrome photograph data to an information processing device;

[2181] means for correcting images on the information processing device and converting monochrome photographs into color;

[2182] A means of correcting the focus of colorized photographs to make them clearer,

[2183] A means of analyzing the user's voice and facial expressions in real time to determine their emotions,

[2184] A means for interacting with a user using generative AI to customize a photo based on the user's instructions and the determined emotions;

[2185] A means to provide users with the final customized photo;

[2186] A system including:

[2187] (Claim ...

Claims

1. A means of photographing or scanning old black-and-white photographs; A means for transmitting the captured or scanned monochrome photograph data to a server; A method to correct images on the server and convert monochrome photos to color, A means of correcting the focus of colorized photographs to make them clearer, A means to interact with users using generative AI and customize photos based on their instructions; A means to provide users with the final customized photo; A system including:

2. 2. The system according to claim 1, wherein color conversion and focus correction of monochrome photographs are performed simultaneously.

3. The system according to claim 1, characterized in that color adjustment and removal of unnecessary background are performed through dialogue between the user and the generating AI.

Images