System

Abstract

A system is provided.SOLUTION: This system includes a means for receiving the video of a camera device in real time, a means for analyzing the received video and detecting the arrangement of an object, the angle of light and color balance, a means for generating advice relating to a photographing technique on the basis of a detection result, a means for displaying the generated advice and a means for automatically adjusting the setting of the camera device on the basis of the advice.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Today, many people desire to take beautiful photographs, but mastering various photography techniques such as composition, exposure, focus, and color balance can be difficult. Mastering these skills is a major challenge, especially for beginners. Traditional photography applications lack the functionality to adequately address these issues, forcing users to manually adjust settings. This makes it difficult to ensure optimal conditions at the moment of capture, resulting in less-than-expected photos. [Means for solving the problem]

[0005] The present invention provides a system for detecting the position of a subject, the angle of light, and the color balance by receiving an image from a camera device in real time and analyzing the image, which specifically includes the following means:

[0006] 1. A means of receiving images from camera equipment in real time

[0007] 2. A means of analyzing received video to detect subject placement, light angle, and color balance

[0008] 3. A means to generate advice on photography techniques based on the detection results

[0009] 4. A way to view the generated advice

[0010] 5. Means for automatically adjusting camera device settings based on advice

[0011] This allows users to receive professional advice in real time while taking photos, and the shooting environment can be optimized by displaying intuitive icons and messages. Furthermore, by providing feedback for editing after taking a photo, the quality of the photo can be further improved. As a result, it is expected that the photography skills of each individual user, from beginners to advanced users, will steadily improve.

[0012] A "camera device" is a device for taking still or video images, and typically includes a smartphone or digital camera.

[0013] "Video" refers to visual information data captured by a camera device, and is usually a series of images divided into frames.

[0014] "Real-time" refers to a state in which data is acquired and processed immediately, and results are output without delay.

[0015] "Means for receiving" refers to a function or device used to obtain data or signals from the outside and process them.

[0016] "Analysis" is the process of examining acquired data in detail and recognizing and judging its components and characteristics.

[0017] A "subject" is an object that appears in a photograph or video, and includes people, objects, landscapes, etc.

[0018] "Positioning" is a concept that describes how subjects and other elements are positioned within an image.

[0019] The "light angle" is the angle that indicates the direction from which light strikes the subject.

[0020] "Color balance" refers to the distribution and harmony of colors within an image, and in particular refers to adjusting white balance and color temperature.

[0021] The "means for generating advice" refers to a function or device that creates specific improvement measures or instructions for shooting based on the analysis results.

[0022] The "display means" refers to a function or device for visually outputting information via a user interface.

[0023] "Means for automatically adjusting settings" refers to a function or device that automatically changes the shooting conditions (exposure, focus, white balance, etc.) of a camera device based on analysis and advice.

[0024] The "visual display means" refers to a function or device that graphically displays the generated advice or feedback on a screen.

[0025] "Exposure" is a factor that determines the brightness of an image captured by a camera device, and is adjusted by the shutter speed, aperture, and ISO sensitivity.

[0026] "Focus" is a function that adjusts the lens position so that the subject is captured clearly.

[0027] "White balance" is a function that adjusts the color tone in photos and videos to the appropriate level, correcting the color tone according to the type of light source. [Brief explanation of the drawings]

[0028] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0029] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0030] First, the terms used in the following description will be explained.

[0031] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0032] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0033] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0034] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0035] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0036] [First embodiment]

[0037] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0038] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0039] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0040] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0041] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0042] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0043] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0044] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0045] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0046] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0047] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0048] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0049] This invention is a system that uses a camera device to analyze video in real time, provide advice on photography, and automatically adjust settings based on the results. This system analyzes video captured from a camera device (e.g., a smartphone camera) to detect the subject's position, lighting angle, and color balance. Based on the results, it generates advice on professional photography techniques and displays it to the user. Furthermore, by automatically adjusting camera settings based on this advice, anyone can easily take high-quality photos.

[0050] System Configuration

[0051] 1. Server

[0052] The server analyzes the images received from the camera in real time and generates advice based on the results. A deep learning model is installed on the server, enabling highly accurate image analysis. Specifically, it has the ability to detect the contours of the subject and calculate the angle of light and color balance. The server also sends the generated advice to devices such as smartphones.

[0053] 2. Terminal

[0054] The device is a smartphone equipped with a camera, and it displays advice from the server to the user and automatically adjusts settings. The device also has the function of sending images captured by the camera to the server. The device also displays the advice received from the server on the user interface, visually conveying it as icons and messages so that the user can intuitively understand it.

[0055] 3. Users

[0056] The user operates the camera and takes pictures according to the system's advice. By adjusting the camera's orientation and subject position based on the icons and messages displayed on the user interface, the user can take high-quality photos.

[0057] Program processing overview (natural language explanation)

[0058] server:

[0059] The server analyzes the video received from the device, analyzing the video frame by frame to detect the contours and position of the subject, the angle of light, and color balance.

[0060] Based on the detected information, it generates specific advice that can be useful when taking a photo, such as "move the subject a little more to the right" or "change the light source."

[0061] The generated advice is sent to the device.

[0062] Device:

[0063] The device displays the advice received from the server on the user interface, allowing the user to adjust the camera direction and subject position.

[0064] It also automatically adjusts camera settings such as exposure, focus, and white balance based on advice from the server.

[0065] User:

[0066] The user adjusts the subject and camera position based on the advice displayed on the device.

[0067] Once you've followed the advice and optimized your composition and settings, simply press the shutter to take the photo, making it easy to capture professional-quality photos.

[0068] Specific examples

[0069] As an example, consider a case where a user takes a portrait photo with a sunset as the background.

[0070] server:

[0071] Detects the position of the setting sun and people, and recognizes backlight conditions.

[0072] It generates advice such as, "Try to change the camera angle a little so the sun is above your subject's head."

[0073] The generated advice is sent to the device.

[0074] Device:

[0075] The user interface displays the advice "Adjust your camera angle" and displays an arrow icon indicating the ideal camera angle.

[0076] Adjust the camera's exposure and change the settings so that the subject is not too dark.

[0077] User:

[0078] Follow the advice displayed on your device to adjust the camera angle and correct the subject's position.

[0079] Press the shutter button to take the picture. As a result, you can take a beautiful photo of the subject, taking advantage of the backlight.

[0080] In this way, the system of the present invention allows users to take high-quality photographs simply and intuitively.

[0081] The processing flow will be explained below.

[0082] Step 1:

[0083] The user launches the camera app on their smartphone, points the camera at the object they want to photograph, and frames the subject on the screen.

[0084] Step 2:

[0085] The device captures video frames from the camera in real time, and the captured video frames are sent to the server at regular intervals.

[0086] Step 3:

[0087] The server analyzes the video frames received from the device and uses deep learning models to detect subject contours, lighting angles, and color balance.

[0088] Step 4:

[0089] The server generates advice based on the analysis results, such as "adjust the camera angle" if the subject is off-center or there is strong backlighting.

[0090] Step 5:

[0091] The server sends the generated advice to the device in the form of a text message or icon.

[0092] Step 6:

[0093] The device displays the advice received from the server on the user interface, with specific instructions and icons displayed on the screen to help the user understand intuitively.

[0094] Step 7:

[0095] The device automatically adjusts camera settings based on advice from the server, including exposure, focus, and white balance. For example, it will correct exposure if backlighting is detected.

[0096] Step 8:

[0097] The user adjusts the subject and camera position according to the advice displayed on the device, and then presses the shutter button to take the photo.

[0098] Step 9:

[0099] Your device will save the captured image and take the ideal photo based on the settings and advice you provided in the previous steps.

[0100] Step 10:

[0101] If necessary, the server will analyze the image after shooting and provide additional feedback, which will be presented to the user as advice for future photography.

[0102] In this way, all steps work together to help users take professional-looking photos with ease.

[0103] Example 1

[0104] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0105] Conventional camera systems require advanced knowledge and experience to master professional photography techniques. Furthermore, editing images and adjusting settings after shooting requires a lot of time and effort, making it difficult for average users to take high-quality photos. There is a demand for a system that solves this problem and allows anyone to easily take high-quality photos.

[0106] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0107] In this invention, the server includes means for receiving video from a camera device in real time, means for analyzing the received video using a deep learning model to detect the subject's position, light angle, and color balance, means for generating advice on professional photography techniques based on the detection results, means for transmitting the generated advice to a terminal, and means for automatically adjusting the settings of the camera device based on the advice displayed on the terminal. This allows users to automatically adjust the camera settings while receiving intuitive advice, making it possible for anyone to easily take high-quality photos.

[0108] "Camera device" refers to a device for capturing images in real time. Examples include smartphones and digital cameras.

[0109] The term "server" refers to a computer device that receives video data sent from a camera device, analyzes it, and generates advice.

[0110] A "deep learning model" refers to an algorithm that uses large amounts of data to learn and accurately analyze features in video. Specific examples include models using Tensorflow (registered trademark) and PyTorch.

[0111] The term "subject" refers to the object captured by the camera, including people, objects, and other things that appear in the image.

[0112] "Light angle" refers to the direction of light shining on the subject, and is an important factor that affects the brightness and shadows of a photograph.

[0113] "Color balance" refers to the arrangement and harmony of colors within an image, and is an element that affects the beauty and naturalness of the entire photograph.

[0114] "Advice" refers to specific instructions or recommendations regarding photography techniques generated by the Server.

[0115] "Terminal" refers to a device equipped with a camera, which displays advice received from the server to the user and adjusts the settings of the camera. Examples include smartphones and tablets.

[0116] The present invention is a system that uses a camera device to analyze video in real time, provides advice on shooting, and automatically adjusts camera settings based on the results. This system is composed of a camera device, a server, and a terminal.

[0117] server:

[0118] The server receives the video sent from the camera in real time and analyzes it using a deep learning model. Specifically, a model using deep learning frameworks such as TensorFlow and PyTorch is installed on the server. This model is used to detect the subject's position, light angle, and color balance. It then generates specific advice on professional photography techniques based on the analysis results. For example, advice such as "Move the subject a little more to the right" or "Change the light source." The generated advice is sent to the device via a REST API.

[0119] Device:

[0120] The terminal is a device (e.g., a smartphone) equipped with a camera, which receives advice sent from the server and displays it on a user interface. Specifically, a message such as "Move the subject a little more to the right" is displayed on the terminal screen, along with appropriate icons and arrows, allowing the user to intuitively understand. The terminal also automatically adjusts camera settings (exposure, focus, white balance, etc.) using a native API based on the advice from the server.

[0121] User:

[0122] The user follows the system's guidance to operate the camera and take high-quality photos. Specifically, they adjust the camera's orientation and subject position according to the advice displayed on the device. Once the composition and settings are optimized, they press the shutter button to take the photo.

[0123] Prompt Sentence Examples

[0124] To aid in the practice of the invention, the following are example prompts to be input to the generative AI model:

[0125] I'm trying to take a portrait with the sunset in the background. How can I capture the subject beautifully in the backlight?

[0126] In this way, the system of the present invention allows users to take high-quality photographs simply and intuitively, and enables anyone to take high-quality photographs quickly, even without professional knowledge or experience.

[0127] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0128] Step 1:

[0129] Video Acquisition:

[0130] The device uses the camera device to capture video in real time. This operation uses the camera API to capture video data frame by frame in JPEG or RAW format. The input is the video data obtained by the user launching the camera app and pointing the camera at a subject, and the output is the video frames converted into digital format.

[0131] Step 2:

[0132] Video transmission:

[0133] The device sends the captured video data to the server via a communication interface. During this process, the data is compressed and encrypted and sent via a REST API. The input is the captured video frame, and the output is the digital data received by the server.

[0134] Step 3:

[0135] Video Analysis:

[0136] The server analyzes the received video data using a deep learning model. Specifically, it uses TensorFlow and PyTorch to detect the subject's contours, position, light angle, color balance, etc. The input is the video data sent from the device, and the output is a set of parameters resulting from the analysis (e.g., subject position coordinates, light source angle, color balance, etc.).

[0137] Step 4:

[0138] Generate advice:

[0139] Based on the video analysis results, the server generates specific advice on professional shooting techniques, such as "move the subject a little more to the right" or "change the light source." The input is a set of parameters from the analysis results, and the output is text data of the generated advice.

[0140] Step 5:

[0141] Send Advice:

[0142] The server sends the generated advice to the device. The data is compressed and encrypted again and sent using the REST API. The input is the text data of the generated advice, and the output is the advice data received by the device.

[0143] Step 6:

[0144] Show Advice:

[0145] The terminal visually displays the advice received from the server on the user interface. Specifically, it conveys the advice to the user using messages and icons. The input is the advice data received from the server, and the output is a visual display that the user can see through the terminal.

[0146] Step 7:

[0147] Adjusting camera settings:

[0148] The device automatically adjusts camera settings such as exposure, focus, and white balance based on advice from the server, using the device's native API. The input is advice data from the server, and the output is the adjusted camera settings.

[0149] Step 8:

[0150] Shooting execution:

[0151] The user adjusts the camera's orientation and subject position according to the advice displayed on the device, and then presses the shutter button to take the photo. The input is the advice displayed on the device, and the output is the high-quality photo taken.

[0152] (Application example 1)

[0153] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0154] Adjusting object placement, light angle, and color balance using conventional camera devices and displays is difficult for users without specialized knowledge, often resulting in low-quality displays and placements. In particular, display placement that maximizes the appeal of products is important in brick-and-mortar stores, and optimal placement must be achieved efficiently and intuitively. To solve this problem, it is necessary to provide a system that allows anyone to easily perform high-quality displays and placements.

[0155] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0156] In this invention, the server includes means for receiving images from a camera device in real time, means for analyzing the received images and detecting object placement, light angle, and color balance, means for generating advice on photography techniques based on the detection results, means for displaying the generated advice, means for automatically adjusting camera device settings based on the advice, means for providing specific advice on object placement and lighting adjustment, and means for generating and displaying advice for optimizing product placement in a store. This enables anyone, even without specialized knowledge, to achieve professional-quality placement and display.

[0157] A "camera device" is a device for capturing video, and includes smartphones, digital cameras, and the like.

[0158] The "means for receiving video in real time" refers to a means for receiving video information transmitted from a camera device without delay and providing it for analysis.

[0159] The "means for detecting the position of an object, the angle of light, and the color balance" refers to a means for identifying the positional relationship of an object in an image, the angle of incidence of light, and the color arrangement state.

[0160] The "means for generating advice on photography techniques" is a means for generating specific advice showing how to take photographs to obtain optimal results, based on the detection results.

[0161] The "means for displaying the generated advice" refers to a means for visually presenting the generated advice to the user.

[0162] The "means for automatically adjusting camera device settings" refers to a means for automatically changing camera settings such as exposure, focus, and white balance based on the advice.

[0163] The "means for providing specific advice regarding the placement of objects and the adjustment of lighting" refers to a means for creating and providing advice regarding the optimization of the placement of objects and lighting in store displays.

[0164] The "means for generating and displaying advice for optimizing product placement in a store" is a means for creating advice for optimizing product placement in a physical store and visually presenting it to the user.

[0165] This invention is a system that uses a camera device to analyze images in real time, provides advice on object placement, lighting angle, and color balance, and automatically adjusts settings based on the results. This system is particularly applicable to optimizing product placement and displays in physical stores.

[0166] The server analyzes the video sent from the smartphone or smart glasses in real time. A deep learning model (such as TensorFlow or PyTorch) is used for the analysis to detect the placement of objects, the angle of light, color balance, etc. Based on the detection results, specific advice is generated, such as "Move the product up 5 cm" or "Dim the lighting a little." This generated advice is sent to the smartphone or smart glasses.

[0167] The device (smartphone or smart glasses) displays the advice received from the server on the user interface. For example, in the case of smart glasses, the advice is projected visually as icons or messages, which the user can use to adjust the display or product placement. The device also captures video data and sends it to the server. It also has the function of automatically adjusting camera settings (exposure, focus, white balance, etc.) based on the advice.

[0168] Users can adjust the placement of objects and lighting according to the advice displayed on their devices. For example, they can rearrange displays in a physical store to maximize the appeal of their products. In this way, professional-quality placement and display can be achieved without specialized knowledge.

[0169] Examples:

[0170] 1. The user wears the smart glasses and transmits the image of the display area to the server.

[0171] 2. The server analyzes the video and generates specific advice such as "Move the display up 5cm."

[0172] 3. The smart glasses provide visual advice to the user, who then adjusts the display accordingly.

[0173] Example prompt sentence:

[0174] "Make sure your product is centered and symmetrical."

[0175] "By dimming the lighting in this area a little, the merchandise will stand out."

[0176] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0177] Step 1:

[0178] The server receives video sent from the terminal in real time. Here, the input is video data from the terminal, and the output is the received video data. Specifically, this is a process in which video captured by the terminal's camera is sent to the server via the Internet.

[0179] Step 2:

[0180] The server analyzes the received video and detects the object placement, light angle, and color balance. The input is the video data received in step 1, and the output is data on the object placement, light angle, and color balance. This process uses a deep learning model (e.g., TensorFlow or PyTorch) to obtain the analysis results.

[0181] Step 3:

[0182] The server generates advice on photography techniques based on the analysis results. The input is the analysis data from step 2, and the output is specific advice information. For example, this process generates advice such as "Move the product up 5 cm" or "Dim the lighting a little."

[0183] Step 4:

[0184] The server sends the generated advice to the terminal. The input is the advice information generated in step 3, and the output is the advice information sent to the terminal. Specifically, this is a process of sending the generated text or icon information to the terminal via the network.

[0185] Step 5:

[0186] The device displays the received advice on the user interface. The input is the advice information received in step 4, and the output is a visual presentation to the user. For example, in the case of smart glasses, this is a process of projecting an icon or message into the user's field of view.

[0187] Step 6:

[0188] The device automatically adjusts the camera settings based on the advice. The input is the advice information received in step 4, and the output is the adjusted camera settings. For example, this is a process of changing exposure, focus, and white balance parameters in real time.

[0189] Step 7:

[0190] The user adjusts the placement of objects and lighting according to the advice displayed on the terminal. The input is the advice information displayed in step 5, and the output is the adjusted placement of objects and lighting. Specifically, this is a process in which the user manually changes the position of products and lighting settings.

[0191] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0192] This invention combines a system that analyzes video in real time using a camera device, provides photography advice, and automatically adjusts settings based on the results with an emotion engine that recognizes the user's emotions. This system analyzes video captured by a camera device (e.g., a smartphone camera) to detect the subject's position, lighting angle, and color balance. Based on the results, it generates advice on professional photography techniques and displays it to the user. Furthermore, the emotion engine recognizes the user's emotional state in real time and adjusts the content and display method of the advice depending on that emotional state. This allows anyone to easily take high-quality photos.

[0193] System Configuration

[0194] 1. Server

[0195] The server analyzes the video received from the camera in real time and generates advice based on the results. It also uses an emotion engine to recognize the user's emotional state and adjusts the advice based on the results. Specifically, it uses a deep learning model to perform highly accurate image analysis, detecting the subject's contours, position, light angle, and color balance. The server then sends the generated advice to devices such as smartphones.

[0196] 2. Terminal

[0197] The device is a smartphone equipped with a camera, and it displays advice from the server to the user and automatically adjusts settings. The device also has the function of sending images captured by the camera to the server. The device also displays the advice received from the server on the user interface, visually conveying it as icons and messages so that the user can intuitively understand it.

[0198] 3. Users

[0199] The user operates the camera and takes pictures according to the system's advice. By adjusting the camera's orientation and subject position based on the icons and messages displayed on the user interface, the user can take high-quality photos. The emotion engine also detects the user's emotional state and provides feedback accordingly.

[0200] Program processing overview (natural language explanation)

[0201] server:

[0202] The server analyzes the video frames received from the device. The video is analyzed frame by frame to detect the contours and position of the subject, the angle of light, and color balance.

[0203] Based on the detection results, it generates specific advice that can be useful when taking photos, such as "move the subject a little more to the right" or "change the light source."

[0204] The emotion engine analyzes the user's emotional state and tailors the advice accordingly: if the user is relaxed, it provides detailed advice, and if the user is tense, it provides simplified advice.

[0205] The generated advice is sent to the device.

[0206] Device:

[0207] The device displays the advice received from the server on the user interface, with specific instructions and icons displayed on the screen to help the user intuitively understand.

[0208] The device adjusts how it displays advice based on the analysis results of the emotion engine: if the user is relaxed, it displays detailed information, and if the user is nervous, it displays simplified information.

[0209] It automatically adjusts camera settings based on advice from the server, including exposure, focus, and white balance. For example, it compensates for exposure if backlighting is detected.

[0210] User:

[0211] The user adjusts the subject and camera position based on the advice displayed on the device.

[0212] Once you've followed the advice and optimized your composition and settings, simply press the shutter button to take the photo, resulting in professional-quality photos with ease.

[0213] After the photo is taken, the server provides additional feedback to help the user further improve their skills.

[0214] Specific examples

[0215] As an example, consider a case where a user takes a portrait photo with a sunset as the background.

[0216] server:

[0217] Detects the position of the setting sun and people, and recognizes backlight conditions.

[0218] It generates advice such as, "Try to change the camera angle a little so the sun is above your subject's head."

[0219] The emotion engine analyzes the user's facial expressions and recognizes that the user is relaxed. In this case, it provides detailed advice.

[0220] The generated advice is sent to the device.

[0221] Device:

[0222] The user interface provides detailed advice such as "Adjust your camera angle" and displays an arrow icon indicating the ideal camera angle.

[0223] Adjust the camera's exposure and change the settings so that the subject is not too dark.

[0224] User:

[0225] Follow the advice displayed on your device to adjust the camera angle and correct the subject's position.

[0226] Press the shutter button to take the picture. As a result, you can take a beautiful photo of the subject, taking advantage of the backlight.

[0227] In this way, the system of the present invention allows users to take high-quality photos simply and intuitively, and further optimizes the photography experience by providing feedback according to the user's emotional state.

[0228] The processing flow will be explained below.

[0229] Step 1:

[0230] The user launches the camera app on their smartphone, points the camera at the subject they want to photograph, and frames the subject on the screen.

[0231] Step 2:

[0232] The device captures video frames from the camera in real time, and the captured video frames are sent to the server at regular intervals.

[0233] Step 3:

[0234] The server analyzes video frames received from the device and uses deep learning models to detect the subject's contours, lighting angle, and color balance, identifying situations where the subject is too close to the center of the screen or where there is strong backlighting.

[0235] Step 4:

[0236] The server generates advice based on the analysis results. For example, if the subject is too close to the center, it generates specific advice such as "move the subject a little more to the right." The server then sends this advice to the device.

[0237] Step 5:

[0238] The device displays the advice received from the server on the user interface. Specific instructions and icons are displayed on the screen so that the user can understand intuitively. For example, the text "Adjust the camera angle" and an arrow icon are displayed.

[0239] Step 6:

[0240] The device automatically adjusts camera settings. For example, if backlighting is detected, it will correct the exposure and change settings to prevent the subject from appearing too dark. It also adjusts the content and display of advice based on the user's emotional state.

[0241] Step 7:

[0242] The server uses an emotion engine to analyze the user's emotional state, analyzing their facial expressions and voice in real time to determine whether they are relaxed or tense.

[0243] Step 8:

[0244] The server then adjusts the advice based on the emotion engine's analysis: if the subject is relaxed, it provides detailed advice, such as "change the direction of the light to reduce shadows," while if the subject is nervous, it provides simplified advice, such as "move the subject a little more to the right."

[0245] Step 9:

[0246] The device displays the adjusted advice received from the server on the user interface, and the user can refer to it to adjust the position of the subject and camera.

[0247] Step 10:

[0248] The user adjusts the subject and camera position according to the advice displayed on the device, and once the adjustments are complete, presses the shutter button to take the photo.

[0249] Step 11:

[0250] Your device will capture and store the image, and your settings and recommendations will help you capture the perfect photo.

[0251] Step 12:

[0252] If necessary, the server will analyze the image after shooting and provide additional feedback, which will be presented to the user as advice for future photography.

[0253] In this way, all steps work together to allow users to easily take professional-looking photos and receive feedback that adapts to their emotional state.

[0254] Example 2

[0255] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0256] Conventional camera systems require users to adjust the composition and settings themselves, requiring the skills and knowledge to take high-quality photos. Furthermore, they are unable to provide appropriate photography advice based on the user's emotional state, resulting in a suboptimal photography experience.

[0257] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0258] In this invention, the server includes means for receiving video from a camera device in real time, means for analyzing the received video and detecting the subject's position, light angle, and color balance, means for generating advice on photography techniques based on the detection results, means for analyzing the user's emotional state and adjusting the content and display method of the advice, means for displaying the generated advice, and means for automatically adjusting the camera device settings based on the advice. This allows users to optimize the photography process through an intuitive interface, making it easy to take high-quality photos. Furthermore, providing feedback according to the user's emotional state improves the photography experience.

[0259] A "camera device" is a device for capturing video, and generally refers to a digital camera, a video camera, a camera mounted on a smartphone, etc.

[0260] "Means for receiving video in real time" refers to a system and protocol for receiving video frames sent from a camera device in real time.

[0261] "Means for analyzing received video" refers to software and hardware that uses image processing technology and machine learning models to detect the subject's position, light angle, color balance, etc. from the received video data.

[0262] "Subject placement" refers to information indicating the position of the subject within the captured video.

[0263] "Light angle" refers to information indicating the angle at which the light source is irradiating the subject in the captured image.

[0264] "Color balance" refers to the distribution and harmony of colors within an image, and in particular the degree to which colors are accurately reproduced under white light.

[0265] "Advice on photography techniques" refers to specific instructions and suggestions that users need to take higher quality photos and videos, based on analyzed video data.

[0266] The "means for displaying the generated advice" refers to a device or interface that allows the user to visually understand the advice, and generally refers to a display or the screen of a mobile device.

[0267] "Means for automatically adjusting camera device settings" refers to software and hardware that allows the system to automatically change settings such as exposure, focus, and white balance of the camera based on the generated advice.

[0268] "Means for analyzing the user's emotional state" refers to software and hardware for analyzing the user's facial expressions, vocal tone, etc., and determining whether the user is relaxed, tense, or otherwise emotional.

[0269] "Means for adjusting the content and display of advice" refers to software and hardware for changing the level of detail and display format of the advice provided based on the user's emotional state.

[0270] "Feedback" refers to evaluations and suggestions for the next shoot provided to the user based on the results of the shoot.

[0271] This system uses a camera device to analyze video in real time, provide advice on shooting, and automatically adjust settings based on the results. It also incorporates an emotion engine that recognizes the user's emotions. This system operates through the interaction of a server, a terminal, and a user.

[0272] Server Roles and Operations

[0273] 1. Receiving video

[0274] The server receives video frames sent from the terminal in real time. To achieve real-time processing, video data is received at a fixed frame rate (e.g., 30 frames per second). It is desirable to use high-performance server hardware.

[0275] 2. Video Analysis

[0276] The server analyzes the received video frames. This analysis uses a deep learning model. Specifically, it applies an image analysis model such as "DeepVision" to detect the subject's contours, position, light angle, and color balance. Specifically, it analyzes the brightness and color components of each pixel in the frame and applies a filter that emphasizes the subject's edges.

[0277] 3. Advice Generation

[0278] Based on the detection results, the system generates specific advice for photography, such as "Moving the subject 20 pixels to the right will improve the composition," and sends this advice in text format to the device.

[0279] 4. Emotion analysis

[0280] The server analyzes the user's facial expression data using emotion recognition software such as "Emotion API." If the user is relaxed, detailed advice is generated, and if the user is tense, simplified advice is generated. Emotion analysis is performed based on the pattern of changes in facial expressions; for example, if the smile detection rate exceeds 80%, the user is judged to be "relaxed."

[0281] 5. Submitting Advice

[0282] The generated advice and how to display it based on emotions are sent to the device in JSON format.

[0283] Terminal roles and processing

[0284] 1. Acquiring footage

[0285] The device acquires video frames from the camera device, and the video is sent to the server in real time. Specifically, the device captures video using the smartphone's camera API and sends it to the server via a streaming protocol.

[0286] 2. Receiving Advice

[0287] The device receives the advice data sent from the server and parses it in JSON format.

[0288] 3. Displaying Advice

[0289] The device displays advice in the user interface, using a UI framework such as "One UI" to display specific instructions and icons on the screen so that users can understand intuitively.

[0290] 4. Automatically adjust camera settings

[0291] The device automatically adjusts the camera settings based on advice from the server. For example, if backlighting is detected, the device uses the exposure compensation function to increase brightness. Specifically, the device uses the camera API to call a function that adjusts the exposure level.

[0292] User Roles and Actions

[0293] 1. Check the advice

[0294] The user checks the advice displayed on the device. The advice is displayed as text and icons, so the content can be understood instantly.

[0295] 2. Camera Controls

[0296] The user adjusts the camera angle and the position of the subject according to the advice provided. For example, the user changes the position of the subject according to the advice "move the subject to the right."

[0297] 3. Shooting

[0298] Once the composition and settings are optimized, the user presses the shutter button to take the photo, resulting in a high-quality photo that reflects the system's advice.

[0299] 4. Review feedback

[0300] After taking a photo, the server provides additional feedback to identify areas for improvement for the next photo. The feedback includes a user's rating of the photo and new advice for the next photo.

[0301] Specific examples

[0302] If a user wants to take a portrait photo with a sunset in the background:

[0303] server:

[0304] The server uses image analysis models to detect the location of the setting sun and people, and recognize backlit conditions.

[0305] It generates advice such as, "Try to change the camera angle a little so the sun is above your subject's head."

[0306] Emotion recognition software analyzes the user's facial expressions and provides detailed advice if they are relaxed.

[0307] The generated advice is sent to the device.

[0308] Device:

[0309] It displays detailed advice such as "Adjust your camera angle" and displays an arrow icon indicating the ideal camera angle.

[0310] Adjust the camera's exposure and change the settings so that the subject is not too dark.

[0311] User:

[0312] Follow the advice displayed on your device to adjust the camera angle and correct the subject's position.

[0313] Press the shutter button to take the picture. As a result, you can take a beautiful photo of the subject, taking advantage of the backlight.

[0314] Prompt Sentence Examples

[0315] "I'm trying to take a portrait with the sunset in the background. I'd like some advice on how to position the camera, taking into account the subject and the angle of the light."

[0316] In this way, the system of the present invention allows users to take high-quality photos simply and intuitively, and further optimizes the photography experience by providing feedback that is responsive to the user's emotional state.

[0317] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0318] Step 1: Acquire footage

[0319] The device acquires video frames from the camera device and transmits them to the server in real time. Specifically, it uses the smartphone's camera API to capture video and transmits the video frames to the server at a constant frame rate (30 frames per second) via a streaming protocol. The input is the video frames acquired from the camera device, and the output is the transmission of real-time video data to the server.

[0320] Step 2: Receiving the video

[0321] The server receives video frames sent from the terminal. Specifically, the server uses a high-performance network interface to receive data in real time. The input is the video frame sent from the terminal, and the output is the video data stored in the server.

[0322] Step 3: Video analysis

[0323] The server analyzes the received video frames. Specifically, it uses the deep learning model "DeepVision" to analyze the brightness and color components of each frame and detect the subject's contours, position, light angle, and color balance. The input is the received video data, and the output is information on the detected subject's position, light angle, and color balance.

[0324] Step 4: Advice Generation

[0325] The server generates specific advice about shooting based on the results of video analysis. Specifically, it generates a text message based on the image analysis results, instructing the user on how to adjust the camera and subject. Specific advice such as "Moving the subject 20 pixels to the right will improve the composition" is generated and output in JSON format. The input is the video analysis results, and the output is the generated advice data.

[0326] Step 5: Sentiment Analysis

[0327] The server analyzes the user's facial expression data. Specifically, it uses emotion recognition software "Emotion API" to analyze the user's emotional state from their facial expressions. It determines whether the user is relaxed or tense, and adjusts the content and display format of the advice accordingly. The input is the user's facial expression data, and the output is the analyzed information on the user's emotional state.

[0328] Step 6: Submitting Advice

[0329] The server sends the generated advice and the emotion analysis results to the terminal. Specifically, it sends the generated advice and the emotion-based display method together in JSON format to the terminal. The input is the generated advice data and the emotion analysis results, and the output is JSON data containing them.

[0330] Step 7: Viewing Advice

[0331] The device analyzes the advice data received from the server and displays it on the user interface. Specifically, it uses "One UI" to display a text message such as "Move the subject to the right" or an arrow icon. The input is the advice data received from the server, and the output is the advice displayed on the user interface.

[0332] Step 8: Automatically adjust camera settings

[0333] The device automatically adjusts the camera settings based on advice from the server. Specifically, it uses the camera API to increase brightness using the exposure compensation function when backlighting is detected, for example. The input is the advice data received from the server, and the output is the automatically adjusted camera settings.

[0334] Step 9: Review the advice

[0335] The user checks the advice displayed on the device and prepares to take the photo. Specific actions include checking the specific instructions and icons displayed on the device screen. The input is the advice displayed on the device, and the output is the user's understanding and preparation.

[0336] Step 10: Shoot

[0337] The user adjusts the camera settings and subject position according to the device's advice, then presses the shutter button to take the photo. Specifically, the user follows the instructions displayed on the device, adjusts the camera angle and subject position, and finally presses the shutter button. The input is the advice displayed on the device and the user's actions, and the output is the high-quality photo taken.

[0338] Step 11: Review feedback

[0339] After taking a photo, the user checks the feedback sent from the server. Specifically, the user checks the evaluation based on the shooting results and suggestions for the next shoot. The input is the feedback data sent from the server after the photo is taken, and the output is the user's understanding and skill improvement for the next shoot.

[0340] (Application example 2)

[0341] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0342] Conventional camera systems require users to have a high level of specialized knowledge to take high-quality photos, and the results are often influenced by the user's emotional state. Therefore, there is a need for a system that allows even general users without specialized knowledge to take high-quality photos intuitively and easily. Furthermore, users who create advertising content, such as influencers, require particularly accurate advice and emotional feedback. Conventional systems do not adequately meet these requirements, and a more user-friendly and advanced system is needed.

[0343] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving video from a camera device in real time, means for analyzing the received video and detecting the subject's position, light angle, and color balance, and means for generating advice on photography techniques based on the detection results. This enables means for displaying the generated advice, means for automatically adjusting the camera device settings based on the advice, means for analyzing the user's emotions and adjusting the advice content based on the user's emotional state, means for displaying specific instructions for photography, and means for providing professional photography advice that optimizes the user's photography experience.

[0344] A "camera device" is a device for receiving and capturing video in real time.

[0345] The "means for analyzing received images" is a technology for detecting the subject's position, light angle, and color balance based on the image data received from the camera device.

[0346] The "means for generating advice" is a method for generating specific instructions or suggestions to assist with photography based on the analysis results.

[0347] The "means for displaying generated advice" is a technique for visually displaying advice as icons or messages so that the user can intuitively understand it.

[0348] The "means for automatically adjusting settings" is a technology for automatically adjusting settings such as exposure, focus, and white balance of a camera based on the generated advice.

[0349] "Means for analyzing user emotions" refers to technology that analyzes the user's emotional state during shooting and provides appropriate feedback based on those emotions.

[0350] The "means for adjusting the content of advice based on the emotional state" is a technique for adjusting the details and display method of advice according to the analyzed emotional state of the user.

[0351] The "means for displaying specific instructions" is a technique for displaying specific operational instructions for shooting in a visually easy-to-understand manner to the user.

[0352] "Means for optimizing the user's photography experience" refers to methods for adjusting various settings and feedback so that the user can take photos in the best overall condition.

[0353] The "means for providing professional photography advice" is a technology for providing appropriate advice to users in real time based on specialized knowledge.

[0354] This invention realizes an influencer photography support application for advertising purposes on a smartphone. A specific embodiment of this system is described below.

[0355] server

[0356] The server has the following features:

[0357] 1. Video reception and analysis:

[0358] It receives video from the camera in real time, and analyzes the video frame by frame to detect subject placement, lighting angle, and color balance using a deep learning framework (e.g., TensorFlow or PyTorch).

[0359] 2. Generating Advice:

[0360] Based on the detection results, it generates specific advice on photography techniques, such as "move the subject a little more to the right" or "change the light source."

[0361] 3. User sentiment analysis:

[0362] It uses an emotion analysis engine (e.g., Emotion API) to recognize the user's emotional state and provide appropriate advice based on the results: detailed advice if the user is relaxed, simple advice if the user is tense.

[0363] 4. Submitting Advice:

[0364] The generated advice is sent to the smartphone device, including feedback based on the user's emotional state.

[0365] Terminal

[0366] The terminal has the following features:

[0367] 1. Video capture and transmission:

[0368] It uses the camera API to capture video and send it to the server in real time.

[0369] 2. Display Advice:

[0370] The advice received from the server is displayed in the user interface, with specific instructions and icons displayed so that the user can intuitively understand.

[0371] 3. Automatically adjust camera settings:

[0372] Automatically adjusts camera settings (exposure, focus, white balance, etc.) based on advice from the server, for example correcting exposure if backlighting is detected.

[0373] User

[0374] The user actions are as follows:

[0375] 1. Shooting according to the advice:

[0376] Follow the advice displayed on your device to adjust the subject and camera position.

[0377] Once you have determined the optimal composition and settings, press the shutter button to take the picture.

[0378] 2. Use feedback:

[0379] After shooting, you can use additional feedback from the server to improve your photography skills.

[0380] Specific examples

[0381] As a concrete example of this system, consider a situation in which a user wants to photograph a new cosmetic product with a sunset as the background. The server analyzes the sunset and the subject's position, recognizes the backlighting, and generates advice such as, "Change the camera angle slightly so that the sun is above the subject's head." If the emotion analysis engine analyzes the user's facial expression and detects a relaxed state, it provides detailed advice. This advice is sent to the device, which displays an arrow icon indicating the ideal angle. The user adjusts the camera angle accordingly and presses the shutter button to take the photo. The result is a professional-quality photo with a beautiful sunset as the background.

[0382] Prompt Sentence Examples

[0383] "To photograph cosmetics against a sunset background, adjust the camera angle and center the subject. Then adjust the lighting to highlight the cosmetics."

[0384] As a result, the present invention provides a system that allows anyone to easily take high-quality advertising photos by providing feedback according to the user's emotional state.

[0385] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0386] Step 1:

[0387] Video acquisition and transmission

[0388] Subject: Terminal

[0389] Specific operation: Captures video in real time through the device's camera API, and sends the video frames captured by the camera to the server at regular intervals.

[0390] Input: Raw video data from the camera.

[0391] Output: Video frames sent to the server in real time.

[0392] Step 2:

[0393] Video reception and analysis

[0394] Subject: Server

[0395] What it does: Receives video sent from the device. Uses a deep learning framework to analyze video frames in real time, specifically detecting subject placement, lighting angle, and color balance.

[0396] Input: Video frames sent from the device.

[0397] Output: Information about subject placement, information about light angle, information about color balance.

[0398] Step 3:

[0399] Generating Advice

[0400] Subject: Server

[0401] Specific actions: Based on the data obtained from video analysis, specific advice on shooting techniques is generated, such as "move the subject to the center" or "change the light source."

[0402] Input: Information about subject placement, light angle, and color balance.

[0403] Output: Specific advice on photography techniques.

[0404] Step 4:

[0405] User sentiment analysis

[0406] Subject: Server

[0407] Specific operation: Using the Emotion API, etc., the system analyzes the user's emotional state from the facial expressions captured by the camera. The system then adjusts the advice accordingly. For example, if the user is relaxed, the system generates detailed advice, and if the user is nervous, the system generates simple advice.

[0408] Input: A video frame containing the user's facial expression.

[0409] Output: Appropriate feedback and advice depending on the user's emotional state.

[0410] Step 5:

[0411] Sending and viewing advice

[0412] Subject: Server, Terminal

[0413] Specific operation: The server sends the generated advice to the device. The device displays the received advice on the user interface, displaying specific instructions and icons that the user can intuitively understand.

[0414] Input: Generated advice,feedback based on emotional state.

[0415] Output: Specific shooting advice and feedback displayed in the user interface.

[0416] Step 6:

[0417] Automatic camera settings adjustment

[0418] Subject: Terminal

[0419] Specific behavior: Automatically adjusts camera settings (e.g., exposure, focus, white balance, etc.) based on advice received from the server. For example, correcting exposure when backlighting is detected.

[0420] Input: The generated advice.

[0421] Output: Camera settings automatically adjusted.

[0422] Step 7:

[0423] User operation and shooting

[0424] Subject: User

[0425] Specific operations: Follow the advice displayed on the device to adjust the position of the subject and camera, obtain the optimal composition and settings, and then press the shutter button to take the photo.

[0426] Input: The advice displayed on the terminal.

[0427] Output: Professional quality photos taken.

[0428] Step 8:

[0429] Post-shoot feedback

[0430] Subject: Server, User

[0431] Specific operation: The server analyzes the captured image and provides additional feedback to the user, who can use this feedback to improve their next capture.

[0432] Input: The photo taken.

[0433] Output: Additional feedback and advice.

[0434] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0435] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0436] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0437] [Second embodiment]

[0438] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0439] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0440] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0441] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0442] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0443] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0444] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0445] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0446] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0447] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0448] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0449] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0450] This invention is a system that uses a camera device to analyze video in real time, provide advice on photography, and automatically adjust settings based on the results. This system analyzes video captured from a camera device (e.g., a smartphone camera) to detect the subject's position, lighting angle, and color balance. Based on the results, it generates advice on professional photography techniques and displays it to the user. Furthermore, by automatically adjusting camera settings based on this advice, anyone can easily take high-quality photos.

[0451] System Configuration

[0452] 1. Server

[0453] The server analyzes the images received from the camera in real time and generates advice based on the results. A deep learning model is installed on the server, enabling highly accurate image analysis. Specifically, it has the ability to detect the contours of the subject and calculate the angle of light and color balance. The server also sends the generated advice to devices such as smartphones.

[0454] 2. Terminal

[0455] The device is a smartphone equipped with a camera, and it displays advice from the server to the user and automatically adjusts settings. The device also has the function of sending images captured by the camera to the server. The device also displays the advice received from the server on the user interface, visually conveying it as icons and messages so that the user can intuitively understand it.

[0456] 3. Users

[0457] The user operates the camera and takes pictures according to the system's advice. By adjusting the camera's orientation and subject position based on the icons and messages displayed on the user interface, the user can take high-quality photos.

[0458] Program processing overview (natural language explanation)

[0459] server:

[0460] The server analyzes the video received from the device, analyzing the video frame by frame to detect the contours and position of the subject, the angle of light, and color balance.

[0461] Based on the detected information, it generates specific advice that can be useful when taking a photo, such as "move the subject a little more to the right" or "change the light source."

[0462] The generated advice is sent to the device.

[0463] Device:

[0464] The device displays the advice received from the server on the user interface, allowing the user to adjust the camera direction and subject position.

[0465] It also automatically adjusts camera settings such as exposure, focus, and white balance based on advice from the server.

[0466] User:

[0467] The user adjusts the subject and camera position based on the advice displayed on the device.

[0468] Once you've followed the advice and optimized your composition and settings, simply press the shutter to take the photo, making it easy to capture professional-quality photos.

[0469] Specific examples

[0470] As an example, consider a case where a user takes a portrait photo with a sunset as the background.

[0471] server:

[0472] Detects the position of the setting sun and people, and recognizes backlight conditions.

[0473] It generates advice such as, "Try to change the camera angle a little so the sun is above your subject's head."

[0474] The generated advice is sent to the device.

[0475] Device:

[0476] The user interface displays the advice "Adjust your camera angle" and displays an arrow icon indicating the ideal camera angle.

[0477] Adjust the camera's exposure and change the settings so that the subject is not too dark.

[0478] User:

[0479] Follow the advice displayed on your device to adjust the camera angle and correct the subject's position.

[0480] Press the shutter button to take the picture. As a result, you can take a beautiful photo of the subject, taking advantage of the backlight.

[0481] In this way, the system of the present invention allows users to take high-quality photographs simply and intuitively.

[0482] The processing flow will be explained below.

[0483] Step 1:

[0484] The user launches the camera app on their smartphone, points the camera at the object they want to photograph, and frames the subject on the screen.

[0485] Step 2:

[0486] The device captures video frames from the camera in real time, and the captured video frames are sent to the server at regular intervals.

[0487] Step 3:

[0488] The server analyzes the video frames received from the device and uses deep learning models to detect subject contours, lighting angles, and color balance.

[0489] Step 4:

[0490] The server generates advice based on the analysis results, such as "adjust the camera angle" if the subject is off-center or there is strong backlighting.

[0491] Step 5:

[0492] The server sends the generated advice to the device in the form of a text message or icon.

[0493] Step 6:

[0494] The device displays the advice received from the server on the user interface, with specific instructions and icons displayed on the screen to help the user understand intuitively.

[0495] Step 7:

[0496] The device automatically adjusts camera settings based on advice from the server, including exposure, focus, and white balance. For example, it will correct exposure if backlighting is detected.

[0497] Step 8:

[0498] The user adjusts the subject and camera position according to the advice displayed on the device, and then presses the shutter button to take the photo.

[0499] Step 9:

[0500] Your device will save the captured image and take the ideal photo based on the settings and advice you provided in the previous steps.

[0501] Step 10:

[0502] If necessary, the server will analyze the image after shooting and provide additional feedback, which will be presented to the user as advice for future photography.

[0503] In this way, all steps work together to help users take professional-looking photos with ease.

[0504] Example 1

[0505] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0506] Conventional camera systems require advanced knowledge and experience to master professional photography techniques. Furthermore, editing images and adjusting settings after shooting requires a lot of time and effort, making it difficult for average users to take high-quality photos. There is a demand for a system that solves this problem and allows anyone to easily take high-quality photos.

[0507] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0508] In this invention, the server includes means for receiving video from a camera device in real time, means for analyzing the received video using a deep learning model to detect the subject's position, light angle, and color balance, means for generating advice on professional photography techniques based on the detection results, means for transmitting the generated advice to a terminal, and means for automatically adjusting the settings of the camera device based on the advice displayed on the terminal. This allows users to automatically adjust the camera settings while receiving intuitive advice, making it possible for anyone to easily take high-quality photos.

[0509] "Camera device" refers to a device for capturing images in real time. Examples include smartphones and digital cameras.

[0510] The term "server" refers to a computer device that receives video data sent from a camera device, analyzes it, and generates advice.

[0511] A "deep learning model" refers to an algorithm that uses large amounts of data to learn and accurately analyze features in video. Specific examples include models using TensorFlow and PyTorch.

[0512] The term "subject" refers to the object captured by the camera, including people, objects, and other things that appear in the image.

[0513] "Light angle" refers to the direction of light shining on the subject, and is an important factor that affects the brightness and shadows of a photograph.

[0514] "Color balance" refers to the arrangement and harmony of colors within an image, and is an element that affects the beauty and naturalness of the entire photograph.

[0515] "Advice" refers to specific instructions or recommendations regarding photography techniques generated by the Server.

[0516] "Terminal" refers to a device equipped with a camera, which displays advice received from the server to the user and adjusts the settings of the camera. Examples include smartphones and tablets.

[0517] The present invention is a system that uses a camera device to analyze video in real time, provides advice on shooting, and automatically adjusts camera settings based on the results. This system is composed of a camera device, a server, and a terminal.

[0518] server:

[0519] The server receives the video sent from the camera in real time and analyzes it using a deep learning model. Specifically, a model using deep learning frameworks such as TensorFlow and PyTorch is installed on the server. This model is used to detect the subject's position, light angle, and color balance. It then generates specific advice on professional photography techniques based on the analysis results. For example, advice such as "Move the subject a little more to the right" or "Change the light source." The generated advice is sent to the device via a REST API.

[0520] Device:

[0521] The terminal is a device (e.g., a smartphone) equipped with a camera, which receives advice sent from the server and displays it on a user interface. Specifically, a message such as "Move the subject a little more to the right" is displayed on the terminal screen, along with appropriate icons and arrows, allowing the user to intuitively understand. The terminal also automatically adjusts camera settings (exposure, focus, white balance, etc.) using a native API based on the advice from the server.

[0522] User:

[0523] The user follows the system's guidance to operate the camera and take high-quality photos. Specifically, they adjust the camera's orientation and subject position according to the advice displayed on the device. Once the composition and settings are optimized, they press the shutter button to take the photo.

[0524] Prompt Sentence Examples

[0525] To aid in the practice of the invention, the following are example prompts to be input to the generative AI model:

[0526] I'm trying to take a portrait with the sunset in the background. How can I capture the subject beautifully in the backlight?

[0527] In this way, the system of the present invention allows users to take high-quality photographs simply and intuitively, and enables anyone to take high-quality photographs quickly, even without professional knowledge or experience.

[0528] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0529] Step 1:

[0530] Video Acquisition:

[0531] The device uses the camera device to capture video in real time. This operation uses the camera API to capture video data frame by frame in JPEG or RAW format. The input is the video data obtained by the user launching the camera app and pointing the camera at a subject, and the output is the video frames converted into digital format.

[0532] Step 2:

[0533] Video transmission:

[0534] The device sends the captured video data to the server via a communication interface. During this process, the data is compressed and encrypted and sent via a REST API. The input is the captured video frame, and the output is the digital data received by the server.

[0535] Step 3:

[0536] Video Analysis:

[0537] The server analyzes the received video data using a deep learning model. Specifically, it uses TensorFlow and PyTorch to detect the subject's contours, position, light angle, color balance, etc. The input is the video data sent from the device, and the output is a set of parameters resulting from the analysis (e.g., subject position coordinates, light source angle, color balance, etc.).

[0538] Step 4:

[0539] Generate advice:

[0540] Based on the video analysis results, the server generates specific advice on professional shooting techniques, such as "move the subject a little more to the right" or "change the light source." The input is a set of parameters from the analysis results, and the output is text data of the generated advice.

[0541] Step 5:

[0542] Send Advice:

[0543] The server sends the generated advice to the device. The data is compressed and encrypted again and sent using the REST API. The input is the text data of the generated advice, and the output is the advice data received by the device.

[0544] Step 6:

[0545] Show Advice:

[0546] The terminal visually displays the advice received from the server on the user interface. Specifically, it conveys the advice to the user using messages and icons. The input is the advice data received from the server, and the output is a visual display that the user can see through the terminal.

[0547] Step 7:

[0548] Adjusting camera settings:

[0549] The device automatically adjusts camera settings such as exposure, focus, and white balance based on advice from the server, using the device's native API. The input is advice data from the server, and the output is the adjusted camera settings.

[0550] Step 8:

[0551] Shooting execution:

[0552] The user adjusts the camera's orientation and subject position according to the advice displayed on the device, and then presses the shutter button to take the photo. The input is the advice displayed on the device, and the output is the high-quality photo taken.

[0553] (Application example 1)

[0554] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0555] Adjusting object placement, light angle, and color balance using conventional camera devices and displays is difficult for users without specialized knowledge, often resulting in low-quality displays and placements. In particular, display placement that maximizes the appeal of products is important in brick-and-mortar stores, and optimal placement must be achieved efficiently and intuitively. To solve this problem, it is necessary to provide a system that allows anyone to easily perform high-quality displays and placements.

[0556] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0557] In this invention, the server includes means for receiving images from a camera device in real time, means for analyzing the received images and detecting object placement, light angle, and color balance, means for generating advice on photography techniques based on the detection results, means for displaying the generated advice, means for automatically adjusting camera device settings based on the advice, means for providing specific advice on object placement and lighting adjustment, and means for generating and displaying advice for optimizing product placement in a store. This enables anyone, even without specialized knowledge, to achieve professional-quality placement and display.

[0558] A "camera device" is a device for capturing video, and includes smartphones, digital cameras, and the like.

[0559] The "means for receiving video in real time" refers to a means for receiving video information transmitted from a camera device without delay and providing it for analysis.

[0560] The "means for detecting the position of an object, the angle of light, and the color balance" refers to a means for identifying the positional relationship of an object in an image, the angle of incidence of light, and the color arrangement state.

[0561] The "means for generating advice on photography techniques" is a means for generating specific advice showing how to take photographs to obtain optimal results, based on the detection results.

[0562] The "means for displaying the generated advice" refers to a means for visually presenting the generated advice to the user.

[0563] The "means for automatically adjusting camera device settings" refers to a means for automatically changing camera settings such as exposure, focus, and white balance based on the advice.

[0564] The "means for providing specific advice regarding the placement of objects and the adjustment of lighting" refers to a means for creating and providing advice regarding the optimization of the placement of objects and lighting in store displays.

[0565] The "means for generating and displaying advice for optimizing product placement in a store" is a means for creating advice for optimizing product placement in a physical store and visually presenting it to the user.

[0566] This invention is a system that uses a camera device to analyze images in real time, provides advice on object placement, lighting angle, and color balance, and automatically adjusts settings based on the results. This system is particularly applicable to optimizing product placement and displays in physical stores.

[0567] The server analyzes the video sent from the smartphone or smart glasses in real time. A deep learning model (such as TensorFlow or PyTorch) is used for the analysis to detect the placement of objects, the angle of light, color balance, etc. Based on the detection results, specific advice is generated, such as "Move the product up 5 cm" or "Dim the lighting a little." This generated advice is sent to the smartphone or smart glasses.

[0568] The device (smartphone or smart glasses) displays the advice received from the server on the user interface. For example, in the case of smart glasses, the advice is projected visually as icons or messages, which the user can use to adjust the display or product placement. The device also captures video data and sends it to the server. It also has the function of automatically adjusting camera settings (exposure, focus, white balance, etc.) based on the advice.

[0569] Users can adjust the placement of objects and lighting according to the advice displayed on their devices. For example, they can rearrange displays in a physical store to maximize the appeal of their products. In this way, professional-quality placement and display can be achieved without specialized knowledge.

[0570] Examples:

[0571] 1. The user wears the smart glasses and transmits the image of the display area to the server.

[0572] 2. The server analyzes the video and generates specific advice such as "Move the display up 5cm."

[0573] 3. The smart glasses provide visual advice to the user, who then adjusts the display accordingly.

[0574] Example prompt sentence:

[0575] "Make sure your product is centered and symmetrical."

[0576] "By dimming the lighting in this area a little, the merchandise will stand out."

[0577] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0578] Step 1:

[0579] The server receives video sent from the terminal in real time. Here, the input is video data from the terminal, and the output is the received video data. Specifically, this is a process in which video captured by the terminal's camera is sent to the server via the Internet.

[0580] Step 2:

[0581] The server analyzes the received video and detects the object placement, light angle, and color balance. The input is the video data received in step 1, and the output is data on the object placement, light angle, and color balance. This process uses a deep learning model (e.g., TensorFlow or PyTorch) to obtain the analysis results.

[0582] Step 3:

[0583] The server generates advice on photography techniques based on the analysis results. The input is the analysis data from step 2, and the output is specific advice information. For example, this process generates advice such as "Move the product up 5 cm" or "Dim the lighting a little."

[0584] Step 4:

[0585] The server sends the generated advice to the terminal. The input is the advice information generated in step 3, and the output is the advice information sent to the terminal. Specifically, this is a process of sending the generated text or icon information to the terminal via the network.

[0586] Step 5:

[0587] The device displays the received advice on the user interface. The input is the advice information received in step 4, and the output is a visual presentation to the user. For example, in the case of smart glasses, this is a process of projecting an icon or message into the user's field of view.

[0588] Step 6:

[0589] The device automatically adjusts the camera settings based on the advice. The input is the advice information received in step 4, and the output is the adjusted camera settings. For example, this is a process of changing exposure, focus, and white balance parameters in real time.

[0590] Step 7:

[0591] The user adjusts the placement of objects and lighting according to the advice displayed on the terminal. The input is the advice information displayed in step 5, and the output is the adjusted placement of objects and lighting. Specifically, this is a process in which the user manually changes the position of products and lighting settings.

[0592] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0593] This invention combines a system that analyzes video in real time using a camera device, provides photography advice, and automatically adjusts settings based on the results with an emotion engine that recognizes the user's emotions. This system analyzes video captured by a camera device (e.g., a smartphone camera) to detect the subject's position, lighting angle, and color balance. Based on the results, it generates advice on professional photography techniques and displays it to the user. Furthermore, the emotion engine recognizes the user's emotional state in real time and adjusts the content and display method of the advice depending on that emotional state. This allows anyone to easily take high-quality photos.

[0594] System Configuration

[0595] 1. Server

[0596] The server analyzes the video received from the camera in real time and generates advice based on the results. It also uses an emotion engine to recognize the user's emotional state and adjusts the advice based on the results. Specifically, it uses a deep learning model to perform highly accurate image analysis, detecting the subject's contours, position, light angle, and color balance. The server then sends the generated advice to devices such as smartphones.

[0597] 2. Terminal

[0598] The device is a smartphone equipped with a camera, and it displays advice from the server to the user and automatically adjusts settings. The device also has the function of sending images captured by the camera to the server. The device also displays the advice received from the server on the user interface, visually conveying it as icons and messages so that the user can intuitively understand it.

[0599] 3. Users

[0600] The user operates the camera and takes pictures according to the system's advice. By adjusting the camera's orientation and subject position based on the icons and messages displayed on the user interface, the user can take high-quality photos. The emotion engine also detects the user's emotional state and provides feedback accordingly.

[0601] Program processing overview (natural language explanation)

[0602] server:

[0603] The server analyzes the video frames received from the device. The video is analyzed frame by frame to detect the contours and position of the subject, the angle of light, and color balance.

[0604] Based on the detection results, it generates specific advice that can be useful when taking photos, such as "move the subject a little more to the right" or "change the light source."

[0605] The emotion engine analyzes the user's emotional state and tailors the advice accordingly: if the user is relaxed, it provides detailed advice, and if the user is tense, it provides simplified advice.

[0606] The generated advice is sent to the device.

[0607] Device:

[0608] The device displays the advice received from the server on the user interface, with specific instructions and icons displayed on the screen to help the user intuitively understand.

[0609] The device adjusts how it displays advice based on the analysis results of the emotion engine: if the user is relaxed, it displays detailed information, and if the user is nervous, it displays simplified information.

[0610] It automatically adjusts camera settings based on advice from the server, including exposure, focus, and white balance. For example, it compensates for exposure if backlighting is detected.

[0611] User:

[0612] The user adjusts the subject and camera position based on the advice displayed on the device.

[0613] Once you've followed the advice and optimized your composition and settings, simply press the shutter button to take the photo, resulting in professional-quality photos with ease.

[0614] After the photo is taken, the server provides additional feedback to help the user further improve their skills.

[0615] Specific examples

[0616] As an example, consider a case where a user takes a portrait photo with a sunset as the background.

[0617] server:

[0618] Detects the position of the setting sun and people, and recognizes backlight conditions.

[0619] It generates advice such as, "Try to change the camera angle a little so the sun is above your subject's head."

[0620] The emotion engine analyzes the user's facial expressions and recognizes that the user is relaxed. In this case, it provides detailed advice.

[0621] The generated advice is sent to the device.

[0622] Device:

[0623] The user interface provides detailed advice such as "Adjust your camera angle" and displays an arrow icon indicating the ideal camera angle.

[0624] Adjust the camera's exposure and change the settings so that the subject is not too dark.

[0625] User:

[0626] Follow the advice displayed on your device to adjust the camera angle and correct the subject's position.

[0627] Press the shutter button to take the picture. As a result, you can take a beautiful photo of the subject, taking advantage of the backlight.

[0628] In this way, the system of the present invention allows users to take high-quality photos simply and intuitively, and further optimizes the photography experience by providing feedback according to the user's emotional state.

[0629] The processing flow will be explained below.

[0630] Step 1:

[0631] The user launches the camera app on their smartphone, points the camera at the subject they want to photograph, and frames the subject on the screen.

[0632] Step 2:

[0633] The device captures video frames from the camera in real time, and the captured video frames are sent to the server at regular intervals.

[0634] Step 3:

[0635] The server analyzes video frames received from the device and uses deep learning models to detect the subject's contours, lighting angle, and color balance, identifying situations where the subject is too close to the center of the screen or where there is strong backlighting.

[0636] Step 4:

[0637] The server generates advice based on the analysis results. For example, if the subject is too close to the center, it generates specific advice such as "move the subject a little more to the right." The server then sends this advice to the device.

[0638] Step 5:

[0639] The device displays the advice received from the server on the user interface. Specific instructions and icons are displayed on the screen so that the user can understand intuitively. For example, the text "Adjust the camera angle" and an arrow icon are displayed.

[0640] Step 6:

[0641] The device automatically adjusts camera settings. For example, if backlighting is detected, it will correct the exposure and change settings to prevent the subject from appearing too dark. It also adjusts the content and display of advice based on the user's emotional state.

[0642] Step 7:

[0643] The server uses an emotion engine to analyze the user's emotional state, analyzing their facial expressions and voice in real time to determine whether they are relaxed or tense.

[0644] Step 8:

[0645] The server then adjusts the advice based on the emotion engine's analysis: if the subject is relaxed, it provides detailed advice, such as "change the direction of the light to reduce shadows," while if the subject is nervous, it provides simplified advice, such as "move the subject a little more to the right."

[0646] Step 9:

[0647] The device displays the adjusted advice received from the server on the user interface, and the user can refer to it to adjust the position of the subject and camera.

[0648] Step 10:

[0649] The user adjusts the subject and camera position according to the advice displayed on the device, and once the adjustments are complete, presses the shutter button to take the photo.

[0650] Step 11:

[0651] Your device will capture and store the image, and your settings and recommendations will help you capture the perfect photo.

[0652] Step 12:

[0653] If necessary, the server will analyze the image after shooting and provide additional feedback, which will be presented to the user as advice for future photography.

[0654] In this way, all steps work together to allow users to easily take professional-looking photos and receive feedback that adapts to their emotional state.

[0655] Example 2

[0656] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0657] Conventional camera systems require users to adjust the composition and settings themselves, requiring the skills and knowledge to take high-quality photos. Furthermore, they are unable to provide appropriate photography advice based on the user's emotional state, resulting in a suboptimal photography experience.

[0658] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0659] In this invention, the server includes means for receiving video from a camera device in real time, means for analyzing the received video and detecting the subject's position, light angle, and color balance, means for generating advice on photography techniques based on the detection results, means for analyzing the user's emotional state and adjusting the content and display method of the advice, means for displaying the generated advice, and means for automatically adjusting the camera device settings based on the advice. This allows users to optimize the photography process through an intuitive interface, making it easy to take high-quality photos. Furthermore, providing feedback according to the user's emotional state improves the photography experience.

[0660] A "camera device" is a device for capturing video, and generally refers to a digital camera, a video camera, a camera mounted on a smartphone, etc.

[0661] "Means for receiving video in real time" refers to a system and protocol for receiving video frames sent from a camera device in real time.

[0662] "Means for analyzing received video" refers to software and hardware that uses image processing technology and machine learning models to detect the subject's position, light angle, color balance, etc. from the received video data.

[0663] "Subject placement" refers to information indicating the position of the subject within the captured video.

[0664] "Light angle" refers to information indicating the angle at which the light source is irradiating the subject in the captured image.

[0665] "Color balance" refers to the distribution and harmony of colors within an image, and in particular the degree to which colors are accurately reproduced under white light.

[0666] "Advice on photography techniques" refers to specific instructions and suggestions that users need to take higher quality photos and videos, based on analyzed video data.

[0667] The "means for displaying the generated advice" refers to a device or interface that allows the user to visually understand the advice, and generally refers to a display or the screen of a mobile device.

[0668] "Means for automatically adjusting camera device settings" refers to software and hardware that allows the system to automatically change settings such as exposure, focus, and white balance of the camera based on the generated advice.

[0669] "Means for analyzing the user's emotional state" refers to software and hardware for analyzing the user's facial expressions, vocal tone, etc., and determining whether the user is relaxed, tense, or otherwise emotional.

[0670] "Means for adjusting the content and display of advice" refers to software and hardware for changing the level of detail and display format of the advice provided based on the user's emotional state.

[0671] "Feedback" refers to evaluations and suggestions for the next shoot provided to the user based on the results of the shoot.

[0672] This system uses a camera device to analyze video in real time, provide advice on shooting, and automatically adjust settings based on the results. It also incorporates an emotion engine that recognizes the user's emotions. This system operates through the interaction of a server, a terminal, and a user.

[0673] Server Roles and Operations

[0674] 1. Receiving video

[0675] The server receives video frames sent from the terminal in real time. To achieve real-time processing, video data is received at a fixed frame rate (e.g., 30 frames per second). It is desirable to use high-performance server hardware.

[0676] 2. Video Analysis

[0677] The server analyzes the received video frames. This analysis uses a deep learning model. Specifically, it applies an image analysis model such as "DeepVision" to detect the subject's contours, position, light angle, and color balance. Specifically, it analyzes the brightness and color components of each pixel in the frame and applies a filter that emphasizes the subject's edges.

[0678] 3. Advice Generation

[0679] Based on the detection results, the system generates specific advice for photography, such as "Moving the subject 20 pixels to the right will improve the composition," and sends this advice in text format to the device.

[0680] 4. Emotion analysis

[0681] The server analyzes the user's facial expression data using emotion recognition software such as "Emotion API." If the user is relaxed, detailed advice is generated, and if the user is tense, simplified advice is generated. Emotion analysis is performed based on the pattern of changes in facial expressions; for example, if the smile detection rate exceeds 80%, the user is judged to be "relaxed."

[0682] 5. Submitting Advice

[0683] The generated advice and how to display it based on emotions are sent to the device in JSON format.

[0684] Terminal roles and processing

[0685] 1. Acquiring footage

[0686] The device acquires video frames from the camera device, and the video is sent to the server in real time. Specifically, the device captures video using the smartphone's camera API and sends it to the server via a streaming protocol.

[0687] 2. Receiving Advice

[0688] The device receives the advice data sent from the server and parses it in JSON format.

[0689] 3. Displaying Advice

[0690] The device displays advice in the user interface, using a UI framework such as "One UI" to display specific instructions and icons on the screen so that users can understand intuitively.

[0691] 4. Automatically adjust camera settings

[0692] The device automatically adjusts the camera settings based on advice from the server. For example, if backlighting is detected, the device uses the exposure compensation function to increase brightness. Specifically, the device uses the camera API to call a function that adjusts the exposure level.

[0693] User Roles and Actions

[0694] 1. Check the advice

[0695] The user checks the advice displayed on the device. The advice is displayed as text and icons, so the content can be understood instantly.

[0696] 2. Camera Controls

[0697] The user adjusts the camera angle and the position of the subject according to the advice provided. For example, the user changes the position of the subject according to the advice "move the subject to the right."

[0698] 3. Shooting

[0699] Once the composition and settings are optimized, the user presses the shutter button to take the photo, resulting in a high-quality photo that reflects the system's advice.

[0700] 4. Review feedback

[0701] After taking a photo, the server provides additional feedback to identify areas for improvement for the next photo. The feedback includes a user's rating of the photo and new advice for the next photo.

[0702] Specific examples

[0703] If a user wants to take a portrait photo with a sunset in the background:

[0704] server:

[0705] The server uses image analysis models to detect the location of the setting sun and people, and recognize backlit conditions.

[0706] It generates advice such as, "Try to change the camera angle a little so the sun is above your subject's head."

[0707] Emotion recognition software analyzes the user's facial expressions and provides detailed advice if they are relaxed.

[0708] The generated advice is sent to the device.

[0709] Device:

[0710] It displays detailed advice such as "Adjust your camera angle" and displays an arrow icon indicating the ideal camera angle.

[0711] Adjust the camera's exposure and change the settings so that the subject is not too dark.

[0712] User:

[0713] Follow the advice displayed on your device to adjust the camera angle and correct the subject's position.

[0714] Press the shutter button to take the picture. As a result, you can take a beautiful photo of the subject, taking advantage of the backlight.

[0715] Prompt Sentence Examples

[0716] "I'm trying to take a portrait with the sunset in the background. I'd like some advice on how to position the camera, taking into account the subject and the angle of the light."

[0717] In this way, the system of the present invention allows users to take high-quality photos simply and intuitively, and further optimizes the photography experience by providing feedback that is responsive to the user's emotional state.

[0718] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0719] Step 1: Acquire footage

[0720] The device acquires video frames from the camera device and transmits them to the server in real time. Specifically, it uses the smartphone's camera API to capture video and transmits the video frames to the server at a constant frame rate (30 frames per second) via a streaming protocol. The input is the video frames acquired from the camera device, and the output is the transmission of real-time video data to the server.

[0721] Step 2: Receiving the video

[0722] The server receives video frames sent from the terminal. Specifically, the server uses a high-performance network interface to receive data in real time. The input is the video frame sent from the terminal, and the output is the video data stored in the server.

[0723] Step 3: Video analysis

[0724] The server analyzes the received video frames. Specifically, it uses the deep learning model "DeepVision" to analyze the brightness and color components of each frame and detect the subject's contours, position, light angle, and color balance. The input is the received video data, and the output is information on the detected subject's position, light angle, and color balance.

[0725] Step 4: Advice Generation

[0726] The server generates specific advice about shooting based on the results of video analysis. Specifically, it generates a text message based on the image analysis results, instructing the user on how to adjust the camera and subject. Specific advice such as "Moving the subject 20 pixels to the right will improve the composition" is generated and output in JSON format. The input is the video analysis results, and the output is the generated advice data.

[0727] Step 5: Sentiment Analysis

[0728] The server analyzes the user's facial expression data. Specifically, it uses emotion recognition software "Emotion API" to analyze the user's emotional state from their facial expressions. It determines whether the user is relaxed or tense, and adjusts the content and display format of the advice accordingly. The input is the user's facial expression data, and the output is the analyzed information on the user's emotional state.

[0729] Step 6: Submitting Advice

[0730] The server sends the generated advice and the emotion analysis results to the terminal. Specifically, it sends the generated advice and the emotion-based display method together in JSON format to the terminal. The input is the generated advice data and the emotion analysis results, and the output is JSON data containing them.

[0731] Step 7: Viewing Advice

[0732] The device analyzes the advice data received from the server and displays it on the user interface. Specifically, it uses "One UI" to display a text message such as "Move the subject to the right" or an arrow icon. The input is the advice data received from the server, and the output is the advice displayed on the user interface.

[0733] Step 8: Automatically adjust camera settings

[0734] The device automatically adjusts the camera settings based on advice from the server. Specifically, it uses the camera API to increase brightness using the exposure compensation function when backlighting is detected, for example. The input is the advice data received from the server, and the output is the automatically adjusted camera settings.

[0735] Step 9: Review the advice

[0736] The user checks the advice displayed on the device and prepares to take the photo. Specific actions include checking the specific instructions and icons displayed on the device screen. The input is the advice displayed on the device, and the output is the user's understanding and preparation.

[0737] Step 10: Shoot

[0738] The user adjusts the camera settings and subject position according to the device's advice, then presses the shutter button to take the photo. Specifically, the user follows the instructions displayed on the device, adjusts the camera angle and subject position, and finally presses the shutter button. The input is the advice displayed on the device and the user's actions, and the output is the high-quality photo taken.

[0739] Step 11: Review feedback

[0740] After taking a photo, the user checks the feedback sent from the server. Specifically, the user checks the evaluation based on the shooting results and suggestions for the next shoot. The input is the feedback data sent from the server after the photo is taken, and the output is the user's understanding and skill improvement for the next shoot.

[0741] (Application example 2)

[0742] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0743] Conventional camera systems require users to have a high level of specialized knowledge to take high-quality photos, and the results are often influenced by the user's emotional state. Therefore, there is a need for a system that allows even general users without specialized knowledge to take high-quality photos intuitively and easily. Furthermore, users who create advertising content, such as influencers, require particularly accurate advice and emotional feedback. Conventional systems do not adequately meet these requirements, and a more user-friendly and advanced system is needed.

[0744] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving video from a camera device in real time, means for analyzing the received video and detecting the subject's position, light angle, and color balance, and means for generating advice on photography techniques based on the detection results. This enables means for displaying the generated advice, means for automatically adjusting the camera device settings based on the advice, means for analyzing the user's emotions and adjusting the advice content based on the user's emotional state, means for displaying specific instructions for photography, and means for providing professional photography advice that optimizes the user's photography experience.

[0745] A "camera device" is a device for receiving and capturing video in real time.

[0746] The "means for analyzing received images" is a technology for detecting the subject's position, light angle, and color balance based on the image data received from the camera device.

[0747] The "means for generating advice" is a method for generating specific instructions or suggestions to assist with photography based on the analysis results.

[0748] The "means for displaying generated advice" is a technique for visually displaying advice as icons or messages so that the user can intuitively understand it.

[0749] The "means for automatically adjusting settings" is a technology for automatically adjusting settings such as exposure, focus, and white balance of a camera based on the generated advice.

[0750] "Means for analyzing user emotions" refers to technology that analyzes the user's emotional state during shooting and provides appropriate feedback based on those emotions.

[0751] The "means for adjusting the content of advice based on the emotional state" is a technique for adjusting the details and display method of advice according to the analyzed emotional state of the user.

[0752] The "means for displaying specific instructions" is a technique for displaying specific operational instructions for shooting in a visually easy-to-understand manner to the user.

[0753] "Means for optimizing the user's photography experience" refers to methods for adjusting various settings and feedback so that the user can take photos in the best overall condition.

[0754] The "means for providing professional photography advice" is a technology for providing appropriate advice to users in real time based on specialized knowledge.

[0755] This invention realizes an influencer photography support application for advertising purposes on a smartphone. A specific embodiment of this system is described below.

[0756] server

[0757] The server has the following features:

[0758] 1. Video reception and analysis:

[0759] It receives video from the camera in real time, and analyzes the video frame by frame to detect subject placement, lighting angle, and color balance using a deep learning framework (e.g., TensorFlow or PyTorch).

[0760] 2. Generating Advice:

[0761] Based on the detection results, it generates specific advice on photography techniques, such as "move the subject a little more to the right" or "change the light source."

[0762] 3. User sentiment analysis:

[0763] It uses an emotion analysis engine (e.g., Emotion API) to recognize the user's emotional state and provide appropriate advice based on the results: detailed advice if the user is relaxed, simple advice if the user is tense.

[0764] 4. Submitting Advice:

[0765] The generated advice is sent to the smartphone device, including feedback based on the user's emotional state.

[0766] Terminal

[0767] The terminal has the following features:

[0768] 1. Video capture and transmission:

[0769] It uses the camera API to capture video and send it to the server in real time.

[0770] 2. Display Advice:

[0771] The advice received from the server is displayed in the user interface, with specific instructions and icons displayed so that the user can intuitively understand.

[0772] 3. Automatically adjust camera settings:

[0773] Automatically adjusts camera settings (exposure, focus, white balance, etc.) based on advice from the server, for example correcting exposure if backlighting is detected.

[0774] User

[0775] The user actions are as follows:

[0776] 1. Shooting according to the advice:

[0777] Follow the advice displayed on your device to adjust the subject and camera position.

[0778] Once you have determined the optimal composition and settings, press the shutter button to take the picture.

[0779] 2. Use feedback:

[0780] After shooting, you can use additional feedback from the server to improve your photography skills.

[0781] Specific examples

[0782] As a concrete example of this system, consider a situation in which a user wants to photograph a new cosmetic product with a sunset as the background. The server analyzes the sunset and the subject's position, recognizes the backlighting, and generates advice such as, "Change the camera angle slightly so that the sun is above the subject's head." If the emotion analysis engine analyzes the user's facial expression and detects a relaxed state, it provides detailed advice. This advice is sent to the device, which displays an arrow icon indicating the ideal angle. The user adjusts the camera angle accordingly and presses the shutter button to take the photo. The result is a professional-quality photo with a beautiful sunset as the background.

[0783] Prompt Sentence Examples

[0784] "To photograph cosmetics against a sunset background, adjust the camera angle and center the subject. Then adjust the lighting to highlight the cosmetics."

[0785] As a result, the present invention provides a system that allows anyone to easily take high-quality advertising photos by providing feedback according to the user's emotional state.

[0786] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0787] Step 1:

[0788] Video acquisition and transmission

[0789] Subject: Terminal

[0790] Specific operation: Captures video in real time through the device's camera API, and sends the video frames captured by the camera to the server at regular intervals.

[0791] Input: Raw video data from the camera.

[0792] Output: Video frames sent to the server in real time.

[0793] Step 2:

[0794] Video reception and analysis

[0795] Subject: Server

[0796] What it does: Receives video sent from the device. Uses a deep learning framework to analyze video frames in real time, specifically detecting subject placement, lighting angle, and color balance.

[0797] Input: Video frames sent from the device.

[0798] Output: Information about subject placement, information about light angle, information about color balance.

[0799] Step 3:

[0800] Generating Advice

[0801] Subject: Server

[0802] Specific actions: Based on the data obtained from video analysis, specific advice on shooting techniques is generated, such as "move the subject to the center" or "change the light source."

[0803] Input: Information about subject placement, light angle, and color balance.

[0804] Output: Specific advice on photography techniques.

[0805] Step 4:

[0806] User sentiment analysis

[0807] Subject: Server

[0808] Specific operation: Using the Emotion API, etc., the system analyzes the user's emotional state from the facial expressions captured by the camera. The system then adjusts the advice accordingly. For example, if the user is relaxed, the system generates detailed advice, and if the user is nervous, the system generates simple advice.

[0809] Input: A video frame containing the user's facial expression.

[0810] Output: Appropriate feedback and advice depending on the user's emotional state.

[0811] Step 5:

[0812] Sending and viewing advice

[0813] Subject: Server, Terminal

[0814] Specific operation: The server sends the generated advice to the device. The device displays the received advice on the user interface, displaying specific instructions and icons that the user can intuitively understand.

[0815] Input: Generated advice,feedback based on emotional state.

[0816] Output: Specific shooting advice and feedback displayed in the user interface.

[0817] Step 6:

[0818] Automatic camera settings adjustment

[0819] Subject: Terminal

[0820] Specific behavior: Automatically adjusts camera settings (e.g., exposure, focus, white balance, etc.) based on advice received from the server. For example, correcting exposure when backlighting is detected.

[0821] Input: The generated advice.

[0822] Output: Camera settings automatically adjusted.

[0823] Step 7:

[0824] User operation and shooting

[0825] Subject: User

[0826] Specific operations: Follow the advice displayed on the device to adjust the position of the subject and camera, obtain the optimal composition and settings, and then press the shutter button to take the photo.

[0827] Input: The advice displayed on the terminal.

[0828] Output: Professional quality photos taken.

[0829] Step 8:

[0830] Post-shoot feedback

[0831] Subject: Server, User

[0832] Specific operation: The server analyzes the captured image and provides additional feedback to the user, who can use this feedback to improve their next capture.

[0833] Input: The photo taken.

[0834] Output: Additional feedback and advice.

[0835] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0836] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0837] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0838] [Third embodiment]

[0839] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0840] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0841] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0842] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0843] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0844] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0845] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0846] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0847] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0848] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0849] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0850] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0851] This invention is a system that uses a camera device to analyze video in real time, provide advice on photography, and automatically adjust settings based on the results. This system analyzes video captured from a camera device (e.g., a smartphone camera) to detect the subject's position, lighting angle, and color balance. Based on the results, it generates advice on professional photography techniques and displays it to the user. Furthermore, by automatically adjusting camera settings based on this advice, anyone can easily take high-quality photos.

[0852] System Configuration

[0853] 1. Server

[0854] The server analyzes the images received from the camera in real time and generates advice based on the results. A deep learning model is installed on the server, enabling highly accurate image analysis. Specifically, it has the ability to detect the contours of the subject and calculate the angle of light and color balance. The server also sends the generated advice to devices such as smartphones.

[0855] 2. Terminal

[0856] The device is a smartphone equipped with a camera, and it displays advice from the server to the user and automatically adjusts settings. The device also has the function of sending images captured by the camera to the server. The device also displays the advice received from the server on the user interface, visually conveying it as icons and messages so that the user can intuitively understand it.

[0857] 3. Users

[0858] The user operates the camera and takes pictures according to the system's advice. By adjusting the camera's orientation and subject position based on the icons and messages displayed on the user interface, the user can take high-quality photos.

[0859] Program processing overview (natural language explanation)

[0860] server:

[0861] The server analyzes the video received from the device, analyzing the video frame by frame to detect the contours and position of the subject, the angle of light, and color balance.

[0862] Based on the detected information, it generates specific advice that can be useful when taking a photo, such as "move the subject a little more to the right" or "change the light source."

[0863] The generated advice is sent to the device.

[0864] Device:

[0865] The device displays the advice received from the server on the user interface, allowing the user to adjust the camera direction and subject position.

[0866] It also automatically adjusts camera settings such as exposure, focus, and white balance based on advice from the server.

[0867] User:

[0868] The user adjusts the subject and camera position based on the advice displayed on the device.

[0869] Once you've followed the advice and optimized your composition and settings, simply press the shutter to take the photo, making it easy to capture professional-quality photos.

[0870] Specific examples

[0871] As an example, consider a case where a user takes a portrait photo with a sunset as the background.

[0872] server:

[0873] Detects the position of the setting sun and people, and recognizes backlight conditions.

[0874] It generates advice such as, "Try to change the camera angle a little so the sun is above your subject's head."

[0875] The generated advice is sent to the device.

[0876] Device:

[0877] The user interface displays the advice "Adjust your camera angle" and displays an arrow icon indicating the ideal camera angle.

[0878] Adjust the camera's exposure and change the settings so that the subject is not too dark.

[0879] User:

[0880] Follow the advice displayed on your device to adjust the camera angle and correct the subject's position.

[0881] Press the shutter button to take the picture. As a result, you can take a beautiful photo of the subject, taking advantage of the backlight.

[0882] In this way, the system of the present invention allows users to take high-quality photographs simply and intuitively.

[0883] The processing flow will be explained below.

[0884] Step 1:

[0885] The user launches the camera app on their smartphone, points the camera at the object they want to photograph, and frames the subject on the screen.

[0886] Step 2:

[0887] The device captures video frames from the camera in real time, and the captured video frames are sent to the server at regular intervals.

[0888] Step 3:

[0889] The server analyzes the video frames received from the device and uses deep learning models to detect subject contours, lighting angles, and color balance.

[0890] Step 4:

[0891] The server generates advice based on the analysis results, such as "adjust the camera angle" if the subject is off-center or there is strong backlighting.

[0892] Step 5:

[0893] The server sends the generated advice to the device in the form of a text message or icon.

[0894] Step 6:

[0895] The device displays the advice received from the server on the user interface, with specific instructions and icons displayed on the screen to help the user understand intuitively.

[0896] Step 7:

[0897] The device automatically adjusts camera settings based on advice from the server, including exposure, focus, and white balance. For example, it will correct exposure if backlighting is detected.

[0898] Step 8:

[0899] The user adjusts the subject and camera position according to the advice displayed on the device, and then presses the shutter button to take the photo.

[0900] Step 9:

[0901] Your device will save the captured image and take the ideal photo based on the settings and advice you provided in the previous steps.

[0902] Step 10:

[0903] If necessary, the server will analyze the image after shooting and provide additional feedback, which will be presented to the user as advice for future photography.

[0904] In this way, all steps work together to help users take professional-looking photos with ease.

[0905] Example 1

[0906] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0907] Conventional camera systems require advanced knowledge and experience to master professional photography techniques. Furthermore, editing images and adjusting settings after shooting requires a lot of time and effort, making it difficult for average users to take high-quality photos. There is a demand for a system that solves this problem and allows anyone to easily take high-quality photos.

[0908] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0909] In this invention, the server includes means for receiving video from a camera device in real time, means for analyzing the received video using a deep learning model to detect the subject's position, light angle, and color balance, means for generating advice on professional photography techniques based on the detection results, means for transmitting the generated advice to a terminal, and means for automatically adjusting the settings of the camera device based on the advice displayed on the terminal. This allows users to automatically adjust the camera settings while receiving intuitive advice, making it possible for anyone to easily take high-quality photos.

[0910] "Camera device" refers to a device for capturing images in real time. Examples include smartphones and digital cameras.

[0911] The term "server" refers to a computer device that receives video data sent from a camera device, analyzes it, and generates advice.

[0912] A "deep learning model" refers to an algorithm that uses large amounts of data to learn and accurately analyze features in video. Specific examples include models using TensorFlow and PyTorch.

[0913] The term "subject" refers to the object captured by the camera, including people, objects, and other things that appear in the image.

[0914] "Light angle" refers to the direction of light shining on the subject, and is an important factor that affects the brightness and shadows of a photograph.

[0915] "Color balance" refers to the arrangement and harmony of colors within an image, and is an element that affects the beauty and naturalness of the entire photograph.

[0916] "Advice" refers to specific instructions or recommendations regarding photography techniques generated by the Server.

[0917] "Terminal" refers to a device equipped with a camera, which displays advice received from the server to the user and adjusts the settings of the camera. Examples include smartphones and tablets.

[0918] The present invention is a system that uses a camera device to analyze video in real time, provides advice on shooting, and automatically adjusts camera settings based on the results. This system is composed of a camera device, a server, and a terminal.

[0919] server:

[0920] The server receives the video sent from the camera in real time and analyzes it using a deep learning model. Specifically, a model using deep learning frameworks such as TensorFlow and PyTorch is installed on the server. This model is used to detect the subject's position, light angle, and color balance. It then generates specific advice on professional photography techniques based on the analysis results. For example, advice such as "Move the subject a little more to the right" or "Change the light source." The generated advice is sent to the device via a REST API.

[0921] Device:

[0922] The terminal is a device (e.g., a smartphone) equipped with a camera, which receives advice sent from the server and displays it on a user interface. Specifically, a message such as "Move the subject a little more to the right" is displayed on the terminal screen, along with appropriate icons and arrows, allowing the user to intuitively understand. The terminal also automatically adjusts camera settings (exposure, focus, white balance, etc.) using a native API based on the advice from the server.

[0923] User:

[0924] The user follows the system's guidance to operate the camera and take high-quality photos. Specifically, they adjust the camera's orientation and subject position according to the advice displayed on the device. Once the composition and settings are optimized, they press the shutter button to take the photo.

[0925] Prompt Sentence Examples

[0926] To aid in the practice of the invention, the following are example prompts to be input to the generative AI model:

[0927] I'm trying to take a portrait with the sunset in the background. How can I capture the subject beautifully in the backlight?

[0928] In this way, the system of the present invention allows users to take high-quality photographs simply and intuitively, and enables anyone to take high-quality photographs quickly, even without professional knowledge or experience.

[0929] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0930] Step 1:

[0931] Video Acquisition:

[0932] The device uses the camera device to capture video in real time. This operation uses the camera API to capture video data frame by frame in JPEG or RAW format. The input is the video data obtained by the user launching the camera app and pointing the camera at a subject, and the output is the video frames converted into digital format.

[0933] Step 2:

[0934] Video transmission:

[0935] The device sends the captured video data to the server via a communication interface. During this process, the data is compressed and encrypted and sent via a REST API. The input is the captured video frame, and the output is the digital data received by the server.

[0936] Step 3:

[0937] Video Analysis:

[0938] The server analyzes the received video data using a deep learning model. Specifically, it uses TensorFlow and PyTorch to detect the subject's contours, position, light angle, color balance, etc. The input is the video data sent from the device, and the output is a set of parameters resulting from the analysis (e.g., subject position coordinates, light source angle, color balance, etc.).

[0939] Step 4:

[0940] Generate advice:

[0941] Based on the video analysis results, the server generates specific advice on professional shooting techniques, such as "move the subject a little more to the right" or "change the light source." The input is a set of parameters from the analysis results, and the output is text data of the generated advice.

[0942] Step 5:

[0943] Send Advice:

[0944] The server sends the generated advice to the device. The data is compressed and encrypted again and sent using the REST API. The input is the text data of the generated advice, and the output is the advice data received by the device.

[0945] Step 6:

[0946] Show Advice:

[0947] The terminal visually displays the advice received from the server on the user interface. Specifically, it conveys the advice to the user using messages and icons. The input is the advice data received from the server, and the output is a visual display that the user can see through the terminal.

[0948] Step 7:

[0949] Adjusting camera settings:

[0950] The device automatically adjusts camera settings such as exposure, focus, and white balance based on advice from the server, using the device's native API. The input is advice data from the server, and the output is the adjusted camera settings.

[0951] Step 8:

[0952] Shooting execution:

[0953] The user adjusts the camera's orientation and subject position according to the advice displayed on the device, and then presses the shutter button to take the photo. The input is the advice displayed on the device, and the output is the high-quality photo taken.

[0954] (Application example 1)

[0955] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0956] Adjusting object placement, light angle, and color balance using conventional camera devices and displays is difficult for users without specialized knowledge, often resulting in low-quality displays and placements. In particular, display placement that maximizes the appeal of products is important in brick-and-mortar stores, and optimal placement must be achieved efficiently and intuitively. To solve this problem, it is necessary to provide a system that allows anyone to easily perform high-quality displays and placements.

[0957] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0958] In this invention, the server includes means for receiving images from a camera device in real time, means for analyzing the received images and detecting object placement, light angle, and color balance, means for generating advice on photography techniques based on the detection results, means for displaying the generated advice, means for automatically adjusting camera device settings based on the advice, means for providing specific advice on object placement and lighting adjustment, and means for generating and displaying advice for optimizing product placement in a store. This enables anyone, even without specialized knowledge, to achieve professional-quality placement and display.

[0959] A "camera device" is a device for capturing video, and includes smartphones, digital cameras, and the like.

[0960] The "means for receiving video in real time" refers to a means for receiving video information transmitted from a camera device without delay and providing it for analysis.

[0961] The "means for detecting the position of an object, the angle of light, and the color balance" refers to a means for identifying the positional relationship of an object in an image, the angle of incidence of light, and the color arrangement state.

[0962] The "means for generating advice on photography techniques" is a means for generating specific advice showing how to take photographs to obtain optimal results, based on the detection results.

[0963] The "means for displaying the generated advice" refers to a means for visually presenting the generated advice to the user.

[0964] The "means for automatically adjusting camera device settings" refers to a means for automatically changing camera settings such as exposure, focus, and white balance based on the advice.

[0965] The "means for providing specific advice regarding the placement of objects and the adjustment of lighting" refers to a means for creating and providing advice regarding the optimization of the placement of objects and lighting in store displays.

[0966] The "means for generating and displaying advice for optimizing product placement in a store" is a means for creating advice for optimizing product placement in a physical store and visually presenting it to the user.

[0967] This invention is a system that uses a camera device to analyze images in real time, provides advice on object placement, lighting angle, and color balance, and automatically adjusts settings based on the results. This system is particularly applicable to optimizing product placement and displays in physical stores.

[0968] The server analyzes the video sent from the smartphone or smart glasses in real time. A deep learning model (such as TensorFlow or PyTorch) is used for the analysis to detect the placement of objects, the angle of light, color balance, etc. Based on the detection results, specific advice is generated, such as "Move the product up 5 cm" or "Dim the lighting a little." This generated advice is sent to the smartphone or smart glasses.

[0969] The device (smartphone or smart glasses) displays the advice received from the server on the user interface. For example, in the case of smart glasses, the advice is projected visually as icons or messages, which the user can use to adjust the display or product placement. The device also captures video data and sends it to the server. It also has the function of automatically adjusting camera settings (exposure, focus, white balance, etc.) based on the advice.

[0970] Users can adjust the placement of objects and lighting according to the advice displayed on their devices. For example, they can rearrange displays in a physical store to maximize the appeal of their products. In this way, professional-quality placement and display can be achieved without specialized knowledge.

[0971] Examples:

[0972] 1. The user wears the smart glasses and transmits the image of the display area to the server.

[0973] 2. The server analyzes the video and generates specific advice such as "Move the display up 5cm."

[0974] 3. The smart glasses provide visual advice to the user, who then adjusts the display accordingly.

[0975] Example prompt sentence:

[0976] "Make sure your product is centered and symmetrical."

[0977] "By dimming the lighting in this area a little, the merchandise will stand out."

[0978] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0979] Step 1:

[0980] The server receives video sent from the terminal in real time. Here, the input is video data from the terminal, and the output is the received video data. Specifically, this is a process in which video captured by the terminal's camera is sent to the server via the Internet.

[0981] Step 2:

[0982] The server analyzes the received video and detects the object placement, light angle, and color balance. The input is the video data received in step 1, and the output is data on the object placement, light angle, and color balance. This process uses a deep learning model (e.g., TensorFlow or PyTorch) to obtain the analysis results.

[0983] Step 3:

[0984] The server generates advice on photography techniques based on the analysis results. The input is the analysis data from step 2, and the output is specific advice information. For example, this process generates advice such as "Move the product up 5 cm" or "Dim the lighting a little."

[0985] Step 4:

[0986] The server sends the generated advice to the terminal. The input is the advice information generated in step 3, and the output is the advice information sent to the terminal. Specifically, this is a process of sending the generated text or icon information to the terminal via the network.

[0987] Step 5:

[0988] The device displays the received advice on the user interface. The input is the advice information received in step 4, and the output is a visual presentation to the user. For example, in the case of smart glasses, this is a process of projecting an icon or message into the user's field of view.

[0989] Step 6:

[0990] The device automatically adjusts the camera settings based on the advice. The input is the advice information received in step 4, and the output is the adjusted camera settings. For example, this is a process of changing exposure, focus, and white balance parameters in real time.

[0991] Step 7:

[0992] The user adjusts the placement of objects and lighting according to the advice displayed on the terminal. The input is the advice information displayed in step 5, and the output is the adjusted placement of objects and lighting. Specifically, this is a process in which the user manually changes the position of products and lighting settings.

[0993] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0994] This invention combines a system that analyzes video in real time using a camera device, provides photography advice, and automatically adjusts settings based on the results with an emotion engine that recognizes the user's emotions. This system analyzes video captured by a camera device (e.g., a smartphone camera) to detect the subject's position, lighting angle, and color balance. Based on the results, it generates advice on professional photography techniques and displays it to the user. Furthermore, the emotion engine recognizes the user's emotional state in real time and adjusts the content and display method of the advice depending on that emotional state. This allows anyone to easily take high-quality photos.

[0995] System Configuration

[0996] 1. Server

[0997] The server analyzes the video received from the camera in real time and generates advice based on the results. It also uses an emotion engine to recognize the user's emotional state and adjusts the advice based on the results. Specifically, it uses a deep learning model to perform highly accurate image analysis, detecting the subject's contours, position, light angle, and color balance. The server then sends the generated advice to devices such as smartphones.

[0998] 2. Terminal

[0999] The device is a smartphone equipped with a camera, and it displays advice from the server to the user and automatically adjusts settings. The device also has the function of sending images captured by the camera to the server. The device also displays the advice received from the server on the user interface, visually conveying it as icons and messages so that the user can intuitively understand it.

[1000] 3. Users

[1001] The user operates the camera and takes pictures according to the system's advice. By adjusting the camera's orientation and subject position based on the icons and messages displayed on the user interface, the user can take high-quality photos. The emotion engine also detects the user's emotional state and provides feedback accordingly.

[1002] Program processing overview (natural language explanation)

[1003] server:

[1004] The server analyzes the video frames received from the device. The video is analyzed frame by frame to detect the contours and position of the subject, the angle of light, and color balance.

[1005] Based on the detection results, it generates specific advice that can be useful when taking photos, such as "move the subject a little more to the right" or "change the light source."

[1006] The emotion engine analyzes the user's emotional state and tailors the advice accordingly: if the user is relaxed, it provides detailed advice, and if the user is tense, it provides simplified advice.

[1007] The generated advice is sent to the device.

[1008] Device:

[1009] The device displays the advice received from the server on the user interface, with specific instructions and icons displayed on the screen to help the user intuitively understand.

[1010] The device adjusts how it displays advice based on the analysis results of the emotion engine: if the user is relaxed, it displays detailed information, and if the user is nervous, it displays simplified information.

[1011] It automatically adjusts camera settings based on advice from the server, including exposure, focus, and white balance. For example, it compensates for exposure if backlighting is detected.

[1012] User:

[1013] The user adjusts the subject and camera position based on the advice displayed on the device.

[1014] Once you've followed the advice and optimized your composition and settings, simply press the shutter button to take the photo, resulting in professional-quality photos with ease.

[1015] After the photo is taken, the server provides additional feedback to help the user further improve their skills.

[1016] Specific examples

[1017] As an example, consider a case where a user takes a portrait photo with a sunset as the background.

[1018] server:

[1019] Detects the position of the setting sun and people, and recognizes backlight conditions.

[1020] It generates advice such as, "Try to change the camera angle a little so the sun is above your subject's head."

[1021] The emotion engine analyzes the user's facial expressions and recognizes that the user is relaxed. In this case, it provides detailed advice.

[1022] The generated advice is sent to the device.

[1023] Device:

[1024] The user interface provides detailed advice such as "Adjust your camera angle" and displays an arrow icon indicating the ideal camera angle.

[1025] Adjust the camera's exposure and change the settings so that the subject is not too dark.

[1026] User:

[1027] Follow the advice displayed on your device to adjust the camera angle and correct the subject's position.

[1028] Press the shutter button to take the picture. As a result, you can take a beautiful photo of the subject, taking advantage of the backlight.

[1029] In this way, the system of the present invention allows users to take high-quality photos simply and intuitively, and further optimizes the photography experience by providing feedback according to the user's emotional state.

[1030] The processing flow will be explained below.

[1031] Step 1:

[1032] The user launches the camera app on their smartphone, points the camera at the subject they want to photograph, and frames the subject on the screen.

[1033] Step 2:

[1034] The device captures video frames from the camera in real time, and the captured video frames are sent to the server at regular intervals.

[1035] Step 3:

[1036] The server analyzes video frames received from the device and uses deep learning models to detect the subject's contours, lighting angle, and color balance, identifying situations where the subject is too close to the center of the screen or where there is strong backlighting.

[1037] Step 4:

[1038] The server generates advice based on the analysis results. For example, if the subject is too close to the center, it generates specific advice such as "move the subject a little more to the right." The server then sends this advice to the device.

[1039] Step 5:

[1040] The device displays the advice received from the server on the user interface. Specific instructions and icons are displayed on the screen so that the user can understand intuitively. For example, the text "Adjust the camera angle" and an arrow icon are displayed.

[1041] Step 6:

[1042] The device automatically adjusts camera settings. For example, if backlighting is detected, it will correct the exposure and change settings to prevent the subject from appearing too dark. It also adjusts the content and display of advice based on the user's emotional state.

[1043] Step 7:

[1044] The server uses an emotion engine to analyze the user's emotional state, analyzing their facial expressions and voice in real time to determine whether they are relaxed or tense.

[1045] Step 8:

[1046] The server then adjusts the advice based on the emotion engine's analysis: if the subject is relaxed, it provides detailed advice, such as "change the direction of the light to reduce shadows," while if the subject is nervous, it provides simplified advice, such as "move the subject a little more to the right."

[1047] Step 9:

[1048] The device displays the adjusted advice received from the server on the user interface, and the user can refer to it to adjust the position of the subject and camera.

[1049] Step 10:

[1050] The user adjusts the subject and camera position according to the advice displayed on the device, and once the adjustments are complete, presses the shutter button to take the photo.

[1051] Step 11:

[1052] Your device will capture and store the image, and your settings and recommendations will help you capture the perfect photo.

[1053] Step 12:

[1054] If necessary, the server will analyze the image after shooting and provide additional feedback, which will be presented to the user as advice for future photography.

[1055] In this way, all steps work together to allow users to easily take professional-looking photos and receive feedback that adapts to their emotional state.

[1056] Example 2

[1057] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1058] Conventional camera systems require users to adjust the composition and settings themselves, requiring the skills and knowledge to take high-quality photos. Furthermore, they are unable to provide appropriate photography advice based on the user's emotional state, resulting in a suboptimal photography experience.

[1059] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1060] In this invention, the server includes means for receiving video from a camera device in real time, means for analyzing the received video and detecting the subject's position, light angle, and color balance, means for generating advice on photography techniques based on the detection results, means for analyzing the user's emotional state and adjusting the content and display method of the advice, means for displaying the generated advice, and means for automatically adjusting the camera device settings based on the advice. This allows users to optimize the photography process through an intuitive interface, making it easy to take high-quality photos. Furthermore, providing feedback according to the user's emotional state improves the photography experience.

[1061] A "camera device" is a device for capturing video, and generally refers to a digital camera, a video camera, a camera mounted on a smartphone, etc.

[1062] "Means for receiving video in real time" refers to a system and protocol for receiving video frames sent from a camera device in real time.

[1063] "Means for analyzing received video" refers to software and hardware that uses image processing technology and machine learning models to detect the subject's position, light angle, color balance, etc. from the received video data.

[1064] "Subject placement" refers to information indicating the position of the subject within the captured video.

[1065] "Light angle" refers to information indicating the angle at which the light source is irradiating the subject in the captured image.

[1066] "Color balance" refers to the distribution and harmony of colors within an image, and in particular the degree to which colors are accurately reproduced under white light.

[1067] "Advice on photography techniques" refers to specific instructions and suggestions that users need to take higher quality photos and videos, based on analyzed video data.

[1068] The "means for displaying the generated advice" refers to a device or interface that allows the user to visually understand the advice, and generally refers to a display or the screen of a mobile device.

[1069] "Means for automatically adjusting camera device settings" refers to software and hardware that allows the system to automatically change settings such as exposure, focus, and white balance of the camera based on the generated advice.

[1070] "Means for analyzing the user's emotional state" refers to software and hardware for analyzing the user's facial expressions, vocal tone, etc., and determining whether the user is relaxed, tense, or otherwise emotional.

[1071] "Means for adjusting the content and display of advice" refers to software and hardware for changing the level of detail and display format of the advice provided based on the user's emotional state.

[1072] "Feedback" refers to evaluations and suggestions for the next shoot provided to the user based on the results of the shoot.

[1073] This system uses a camera device to analyze video in real time, provide advice on shooting, and automatically adjust settings based on the results. It also incorporates an emotion engine that recognizes the user's emotions. This system operates through the interaction of a server, a terminal, and a user.

[1074] Server Roles and Operations

[1075] 1. Receiving video

[1076] The server receives video frames sent from the terminal in real time. To achieve real-time processing, video data is received at a fixed frame rate (e.g., 30 frames per second). It is desirable to use high-performance server hardware.

[1077] 2. Video Analysis

[1078] The server analyzes the received video frames. This analysis uses a deep learning model. Specifically, it applies an image analysis model such as "DeepVision" to detect the subject's contours, position, light angle, and color balance. Specifically, it analyzes the brightness and color components of each pixel in the frame and applies a filter that emphasizes the subject's edges.

[1079] 3. Advice Generation

[1080] Based on the detection results, the system generates specific advice for photography, such as "Moving the subject 20 pixels to the right will improve the composition," and sends this advice in text format to the device.

[1081] 4. Emotion analysis

[1082] The server analyzes the user's facial expression data using emotion recognition software such as "Emotion API." If the user is relaxed, detailed advice is generated, and if the user is tense, simplified advice is generated. Emotion analysis is performed based on the pattern of changes in facial expressions; for example, if the smile detection rate exceeds 80%, the user is judged to be "relaxed."

[1083] 5. Submitting Advice

[1084] The generated advice and how to display it based on emotions are sent to the device in JSON format.

[1085] Terminal roles and processing

[1086] 1. Acquiring footage

[1087] The device acquires video frames from the camera device, and the video is sent to the server in real time. Specifically, the device captures video using the smartphone's camera API and sends it to the server via a streaming protocol.

[1088] 2. Receiving Advice

[1089] The device receives the advice data sent from the server and parses it in JSON format.

[1090] 3. Displaying Advice

[1091] The device displays advice in the user interface, using a UI framework such as "One UI" to display specific instructions and icons on the screen so that users can understand intuitively.

[1092] 4. Automatically adjust camera settings

[1093] The device automatically adjusts the camera settings based on advice from the server. For example, if backlighting is detected, the device uses the exposure compensation function to increase brightness. Specifically, the device uses the camera API to call a function that adjusts the exposure level.

[1094] User Roles and Actions

[1095] 1. Check the advice

[1096] The user checks the advice displayed on the device. The advice is displayed as text and icons, so the content can be understood instantly.

[1097] 2. Camera Controls

[1098] The user adjusts the camera angle and the position of the subject according to the advice provided. For example, the user changes the position of the subject according to the advice "move the subject to the right."

[1099] 3. Shooting

[1100] Once the composition and settings are optimized, the user presses the shutter button to take the photo, resulting in a high-quality photo that reflects the system's advice.

[1101] 4. Review feedback

[1102] After taking a photo, the server provides additional feedback to identify areas for improvement for the next photo. The feedback includes a user's rating of the photo and new advice for the next photo.

[1103] Specific examples

[1104] If a user wants to take a portrait photo with a sunset in the background:

[1105] server:

[1106] The server uses image analysis models to detect the location of the setting sun and people, and recognize backlit conditions.

[1107] It generates advice such as, "Try to change the camera angle a little so the sun is above your subject's head."

[1108] Emotion recognition software analyzes the user's facial expressions and provides detailed advice if they are relaxed.

[1109] The generated advice is sent to the device.

[1110] Device:

[1111] It displays detailed advice such as "Adjust your camera angle" and displays an arrow icon indicating the ideal camera angle.

[1112] Adjust the camera's exposure and change the settings so that the subject is not too dark.

[1113] User:

[1114] Follow the advice displayed on your device to adjust the camera angle and correct the subject's position.

[1115] Press the shutter button to take the picture. As a result, you can take a beautiful photo of the subject, taking advantage of the backlight.

[1116] Prompt Sentence Examples

[1117] "I'm trying to take a portrait with the sunset in the background. I'd like some advice on how to position the camera, taking into account the subject and the angle of the light."

[1118] In this way, the system of the present invention allows users to take high-quality photos simply and intuitively, and further optimizes the photography experience by providing feedback that is responsive to the user's emotional state.

[1119] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1120] Step 1: Acquire footage

[1121] The device acquires video frames from the camera device and transmits them to the server in real time. Specifically, it uses the smartphone's camera API to capture video and transmits the video frames to the server at a constant frame rate (30 frames per second) via a streaming protocol. The input is the video frames acquired from the camera device, and the output is the transmission of real-time video data to the server.

[1122] Step 2: Receiving the video

[1123] The server receives video frames sent from the terminal. Specifically, the server uses a high-performance network interface to receive data in real time. The input is the video frame sent from the terminal, and the output is the video data stored in the server.

[1124] Step 3: Video analysis

[1125] The server analyzes the received video frames. Specifically, it uses the deep learning model "DeepVision" to analyze the brightness and color components of each frame and detect the subject's contours, position, light angle, and color balance. The input is the received video data, and the output is information on the detected subject's position, light angle, and color balance.

[1126] Step 4: Advice Generation

[1127] The server generates specific advice about shooting based on the results of video analysis. Specifically, it generates a text message based on the image analysis results, instructing the user on how to adjust the camera and subject. Specific advice such as "Moving the subject 20 pixels to the right will improve the composition" is generated and output in JSON format. The input is the video analysis results, and the output is the generated advice data.

[1128] Step 5: Sentiment Analysis

[1129] The server analyzes the user's facial expression data. Specifically, it uses emotion recognition software "Emotion API" to analyze the user's emotional state from their facial expressions. It determines whether the user is relaxed or tense, and adjusts the content and display format of the advice accordingly. The input is the user's facial expression data, and the output is the analyzed information on the user's emotional state.

[1130] Step 6: Submitting Advice

[1131] The server sends the generated advice and the emotion analysis results to the terminal. Specifically, it sends the generated advice and the emotion-based display method together in JSON format to the terminal. The input is the generated advice data and the emotion analysis results, and the output is JSON data containing them.

[1132] Step 7: Viewing Advice

[1133] The device analyzes the advice data received from the server and displays it on the user interface. Specifically, it uses "One UI" to display a text message such as "Move the subject to the right" or an arrow icon. The input is the advice data received from the server, and the output is the advice displayed on the user interface.

[1134] Step 8: Automatically adjust camera settings

[1135] The device automatically adjusts the camera settings based on advice from the server. Specifically, it uses the camera API to increase brightness using the exposure compensation function when backlighting is detected, for example. The input is the advice data received from the server, and the output is the automatically adjusted camera settings.

[1136] Step 9: Review the advice

[1137] The user checks the advice displayed on the device and prepares to take the photo. Specific actions include checking the specific instructions and icons displayed on the device screen. The input is the advice displayed on the device, and the output is the user's understanding and preparation.

[1138] Step 10: Shoot

[1139] The user adjusts the camera settings and subject position according to the device's advice, then presses the shutter button to take the photo. Specifically, the user follows the instructions displayed on the device, adjusts the camera angle and subject position, and finally presses the shutter button. The input is the advice displayed on the device and the user's actions, and the output is the high-quality photo taken.

[1140] Step 11: Review feedback

[1141] After taking a photo, the user checks the feedback sent from the server. Specifically, the user checks the evaluation based on the shooting results and suggestions for the next shoot. The input is the feedback data sent from the server after the photo is taken, and the output is the user's understanding and skill improvement for the next shoot.

[1142] (Application example 2)

[1143] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1144] Conventional camera systems require users to have a high level of specialized knowledge to take high-quality photos, and the results are often influenced by the user's emotional state. Therefore, there is a need for a system that allows even general users without specialized knowledge to take high-quality photos intuitively and easily. Furthermore, users who create advertising content, such as influencers, require particularly accurate advice and emotional feedback. Conventional systems do not adequately meet these requirements, and a more user-friendly and advanced system is needed.

[1145] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving video from a camera device in real time, means for analyzing the received video and detecting the subject's position, light angle, and color balance, and means for generating advice on photography techniques based on the detection results. This enables means for displaying the generated advice, means for automatically adjusting the camera device settings based on the advice, means for analyzing the user's emotions and adjusting the advice content based on the user's emotional state, means for displaying specific instructions for photography, and means for providing professional photography advice that optimizes the user's photography experience.

[1146] A "camera device" is a device for receiving and capturing video in real time.

[1147] The "means for analyzing received images" is a technology for detecting the subject's position, light angle, and color balance based on the image data received from the camera device.

[1148] The "means for generating advice" is a method for generating specific instructions or suggestions to assist with photography based on the analysis results.

[1149] The "means for displaying generated advice" is a technique for visually displaying advice as icons or messages so that the user can intuitively understand it.

[1150] The "means for automatically adjusting settings" is a technology for automatically adjusting settings such as exposure, focus, and white balance of a camera based on the generated advice.

[1151] "Means for analyzing user emotions" refers to technology that analyzes the user's emotional state during shooting and provides appropriate feedback based on those emotions.

[1152] The "means for adjusting the content of advice based on the emotional state" is a technique for adjusting the details and display method of advice according to the analyzed emotional state of the user.

[1153] The "means for displaying specific instructions" is a technique for displaying specific operational instructions for shooting in a visually easy-to-understand manner to the user.

[1154] "Means for optimizing the user's photography experience" refers to methods for adjusting various settings and feedback so that the user can take photos in the best overall condition.

[1155] The "means for providing professional photography advice" is a technology for providing appropriate advice to users in real time based on specialized knowledge.

[1156] This invention realizes an influencer photography support application for advertising purposes on a smartphone. A specific embodiment of this system is described below.

[1157] server

[1158] The server has the following features:

[1159] 1. Video reception and analysis:

[1160] It receives video from the camera in real time, and analyzes the video frame by frame to detect subject placement, lighting angle, and color balance using a deep learning framework (e.g., TensorFlow or PyTorch).

[1161] 2. Generating Advice:

[1162] Based on the detection results, it generates specific advice on photography techniques, such as "move the subject a little more to the right" or "change the light source."

[1163] 3. User sentiment analysis:

[1164] It uses an emotion analysis engine (e.g., Emotion API) to recognize the user's emotional state and provide appropriate advice based on the results: detailed advice if the user is relaxed, simple advice if the user is tense.

[1165] 4. Submitting Advice:

[1166] The generated advice is sent to the smartphone device, including feedback based on the user's emotional state.

[1167] Terminal

[1168] The terminal has the following features:

[1169] 1. Video capture and transmission:

[1170] It uses the camera API to capture video and send it to the server in real time.

[1171] 2. Display Advice:

[1172] The advice received from the server is displayed in the user interface, with specific instructions and icons displayed so that the user can intuitively understand.

[1173] 3. Automatically adjust camera settings:

[1174] Automatically adjusts camera settings (exposure, focus, white balance, etc.) based on advice from the server, for example correcting exposure if backlighting is detected.

[1175] User

[1176] The user actions are as follows:

[1177] 1. Shooting according to the advice:

[1178] Follow the advice displayed on your device to adjust the subject and camera position.

[1179] Once you have determined the optimal composition and settings, press the shutter button to take the picture.

[1180] 2. Use feedback:

[1181] After shooting, you can use additional feedback from the server to improve your photography skills.

[1182] Specific examples

[1183] As a concrete example of this system, consider a situation in which a user wants to photograph a new cosmetic product with a sunset as the background. The server analyzes the sunset and the subject's position, recognizes the backlighting, and generates advice such as, "Change the camera angle slightly so that the sun is above the subject's head." If the emotion analysis engine analyzes the user's facial expression and detects a relaxed state, it provides detailed advice. This advice is sent to the device, which displays an arrow icon indicating the ideal angle. The user adjusts the camera angle accordingly and presses the shutter button to take the photo. The result is a professional-quality photo with a beautiful sunset as the background.

[1184] Prompt Sentence Examples

[1185] "To photograph cosmetics against a sunset background, adjust the camera angle and center the subject. Then adjust the lighting to highlight the cosmetics."

[1186] As a result, the present invention provides a system that allows anyone to easily take high-quality advertising photos by providing feedback according to the user's emotional state.

[1187] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1188] Step 1:

[1189] Video acquisition and transmission

[1190] Subject: Terminal

[1191] Specific operation: Captures video in real time through the device's camera API, and sends the video frames captured by the camera to the server at regular intervals.

[1192] Input: Raw video data from the camera.

[1193] Output: Video frames sent to the server in real time.

[1194] Step 2:

[1195] Video reception and analysis

[1196] Subject: Server

[1197] What it does: Receives video sent from the device. Uses a deep learning framework to analyze video frames in real time, specifically detecting subject placement, lighting angle, and color balance.

[1198] Input: Video frames sent from the device.

[1199] Output: Information about subject placement, information about light angle, information about color balance.

[1200] Step 3:

[1201] Generating Advice

[1202] Subject: Server

[1203] Specific actions: Based on the data obtained from video analysis, specific advice on shooting techniques is generated, such as "move the subject to the center" or "change the light source."

[1204] Input: Information about subject placement, light angle, and color balance.

[1205] Output: Specific advice on photography techniques.

[1206] Step 4:

[1207] User sentiment analysis

[1208] Subject: Server

[1209] Specific operation: Using the Emotion API, etc., the system analyzes the user's emotional state from the facial expressions captured by the camera. The system then adjusts the advice accordingly. For example, if the user is relaxed, the system generates detailed advice, and if the user is nervous, the system generates simple advice.

[1210] Input: A video frame containing the user's facial expression.

[1211] Output: Appropriate feedback and advice depending on the user's emotional state.

[1212] Step 5:

[1213] Sending and viewing advice

[1214] Subject: Server, Terminal

[1215] Specific operation: The server sends the generated advice to the device. The device displays the received advice on the user interface, displaying specific instructions and icons that the user can intuitively understand.

[1216] Input: Generated advice,feedback based on emotional state.

[1217] Output: Specific shooting advice and feedback displayed in the user interface.

[1218] Step 6:

[1219] Automatic camera settings adjustment

[1220] Subject: Terminal

[1221] Specific behavior: Automatically adjusts camera settings (e.g., exposure, focus, white balance, etc.) based on advice received from the server. For example, correcting exposure when backlighting is detected.

[1222] Input: The generated advice.

[1223] Output: Camera settings automatically adjusted.

[1224] Step 7:

[1225] User operation and shooting

[1226] Subject: User

[1227] Specific operations: Follow the advice displayed on the device to adjust the position of the subject and camera, obtain the optimal composition and settings, and then press the shutter button to take the photo.

[1228] Input: The advice displayed on the terminal.

[1229] Output: Professional quality photos taken.

[1230] Step 8:

[1231] Post-shoot feedback

[1232] Subject: Server, User

[1233] Specific operation: The server analyzes the captured image and provides additional feedback to the user, who can use this feedback to improve their next capture.

[1234] Input: The photo taken.

[1235] Output: Additional feedback and advice.

[1236] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1237] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1238] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1239] [Fourth embodiment]

[1240] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1241] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1242] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1243] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1244] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1245] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1246] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1247] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1248] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1249] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1250] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1251] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1252] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1253] This invention is a system that uses a camera device to analyze video in real time, provide advice on photography, and automatically adjust settings based on the results. This system analyzes video captured from a camera device (e.g., a smartphone camera) to detect the subject's position, lighting angle, and color balance. Based on the results, it generates advice on professional photography techniques and displays it to the user. Furthermore, by automatically adjusting camera settings based on this advice, anyone can easily take high-quality photos.

[1254] System Configuration

[1255] 1. Server

[1256] The server analyzes the images received from the camera in real time and generates advice based on the results. A deep learning model is installed on the server, enabling highly accurate image analysis. Specifically, it has the ability to detect the contours of the subject and calculate the angle of light and color balance. The server also sends the generated advice to devices such as smartphones.

[1257] 2. Terminal

[1258] The device is a smartphone equipped with a camera, and it displays advice from the server to the user and automatically adjusts settings. The device also has the function of sending images captured by the camera to the server. The device also displays the advice received from the server on the user interface, visually conveying it as icons and messages so that the user can intuitively understand it.

[1259] 3. Users

[1260] The user operates the camera and takes pictures according to the system's advice. By adjusting the camera's orientation and subject position based on the icons and messages displayed on the user interface, the user can take high-quality photos.

[1261] Program processing overview (natural language explanation)

[1262] server:

[1263] The server analyzes the video received from the device, analyzing the video frame by frame to detect the contours and position of the subject, the angle of light, and color balance.

[1264] Based on the detected information, it generates specific advice that can be useful when taking a photo, such as "move the subject a little more to the right" or "change the light source."

[1265] The generated advice is sent to the device.

[1266] Device:

[1267] The device displays the advice received from the server on the user interface, allowing the user to adjust the camera direction and subject position.

[1268] It also automatically adjusts camera settings such as exposure, focus, and white balance based on advice from the server.

[1269] User:

[1270] The user adjusts the subject and camera position based on the advice displayed on the device.

[1271] Once you've followed the advice and optimized your composition and settings, simply press the shutter to take the photo, making it easy to capture professional-quality photos.

[1272] Specific examples

[1273] As an example, consider a case where a user takes a portrait photo with a sunset as the background.

[1274] server:

[1275] Detects the position of the setting sun and people, and recognizes backlight conditions.

[1276] It generates advice such as, "Try to change the camera angle a little so the sun is above your subject's head."

[1277] The generated advice is sent to the device.

[1278] Device:

[1279] The user interface displays the advice "Adjust your camera angle" and displays an arrow icon indicating the ideal camera angle.

[1280] Adjust the camera's exposure and change the settings so that the subject is not too dark.

[1281] User:

[1282] Follow the advice displayed on your device to adjust the camera angle and correct the subject's position.

[1283] Press the shutter button to take the picture. As a result, you can take a beautiful photo of the subject, taking advantage of the backlight.

[1284] In this way, the system of the present invention allows users to take high-quality photographs simply and intuitively.

[1285] The processing flow will be explained below.

[1286] Step 1:

[1287] The user launches the camera app on their smartphone, points the camera at the object they want to photograph, and frames the subject on the screen.

[1288] Step 2:

[1289] The device captures video frames from the camera in real time, and the captured video frames are sent to the server at regular intervals.

[1290] Step 3:

[1291] The server analyzes the video frames received from the device and uses deep learning models to detect subject contours, lighting angles, and color balance.

[1292] Step 4:

[1293] The server generates advice based on the analysis results, such as "adjust the camera angle" if the subject is off-center or there is strong backlighting.

[1294] Step 5:

[1295] The server sends the generated advice to the device in the form of a text message or icon.

[1296] Step 6:

[1297] The device displays the advice received from the server on the user interface, with specific instructions and icons displayed on the screen to help the user understand intuitively.

[1298] Step 7:

[1299] The device automatically adjusts camera settings based on advice from the server, including exposure, focus, and white balance. For example, it will correct exposure if backlighting is detected.

[1300] Step 8:

[1301] The user adjusts the subject and camera position according to the advice displayed on the device, and then presses the shutter button to take the photo.

[1302] Step 9:

[1303] Your device will save the captured image and take the ideal photo based on the settings and advice you provided in the previous steps.

[1304] Step 10:

[1305] If necessary, the server will analyze the image after shooting and provide additional feedback, which will be presented to the user as advice for future photography.

[1306] In this way, all steps work together to help users take professional-looking photos with ease.

[1307] Example 1

[1308] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1309] Conventional camera systems require advanced knowledge and experience to master professional photography techniques. Furthermore, editing images and adjusting settings after shooting requires a lot of time and effort, making it difficult for average users to take high-quality photos. There is a demand for a system that solves this problem and allows anyone to easily take high-quality photos.

[1310] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1311] In this invention, the server includes means for receiving video from a camera device in real time, means for analyzing the received video using a deep learning model to detect the subject's position, light angle, and color balance, means for generating advice on professional photography techniques based on the detection results, means for transmitting the generated advice to a terminal, and means for automatically adjusting the settings of the camera device based on the advice displayed on the terminal. This allows users to automatically adjust the camera settings while receiving intuitive advice, making it possible for anyone to easily take high-quality photos.

[1312] "Camera device" refers to a device for capturing images in real time. Examples include smartphones and digital cameras.

[1313] The term "server" refers to a computer device that receives video data sent from a camera device, analyzes it, and generates advice.

[1314] A "deep learning model" refers to an algorithm that uses large amounts of data to learn and accurately analyze features in video. Specific examples include models using TensorFlow and PyTorch.

[1315] The term "subject" refers to the object captured by the camera, including people, objects, and other things that appear in the image.

[1316] "Light angle" refers to the direction of light shining on the subject, and is an important factor that affects the brightness and shadows of a photograph.

[1317] "Color balance" refers to the arrangement and harmony of colors within an image, and is an element that affects the beauty and naturalness of the entire photograph.

[1318] "Advice" refers to specific instructions or recommendations regarding photography techniques generated by the Server.

[1319] "Terminal" refers to a device equipped with a camera, which displays advice received from the server to the user and adjusts the settings of the camera. Examples include smartphones and tablets.

[1320] The present invention is a system that uses a camera device to analyze video in real time, provides advice on shooting, and automatically adjusts camera settings based on the results. This system is composed of a camera device, a server, and a terminal.

[1321] server:

[1322] The server receives the video sent from the camera in real time and analyzes it using a deep learning model. Specifically, a model using deep learning frameworks such as TensorFlow and PyTorch is installed on the server. This model is used to detect the subject's position, light angle, and color balance. It then generates specific advice on professional photography techniques based on the analysis results. For example, advice such as "Move the subject a little more to the right" or "Change the light source." The generated advice is sent to the device via a REST API.

[1323] Device:

[1324] The terminal is a device (e.g., a smartphone) equipped with a camera, which receives advice sent from the server and displays it on a user interface. Specifically, a message such as "Move the subject a little more to the right" is displayed on the terminal screen, along with appropriate icons and arrows, allowing the user to intuitively understand. The terminal also automatically adjusts camera settings (exposure, focus, white balance, etc.) using a native API based on the advice from the server.

[1325] User:

[1326] The user follows the system's guidance to operate the camera and take high-quality photos. Specifically, they adjust the camera's orientation and subject position according to the advice displayed on the device. Once the composition and settings are optimized, they press the shutter button to take the photo.

[1327] Prompt Sentence Examples

[1328] To aid in the practice of the invention, the following are example prompts to be input to the generative AI model:

[1329] I'm trying to take a portrait with the sunset in the background. How can I capture the subject beautifully in the backlight?

[1330] In this way, the system of the present invention allows users to take high-quality photographs simply and intuitively, and enables anyone to take high-quality photographs quickly, even without professional knowledge or experience.

[1331] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1332] Step 1:

[1333] Video Acquisition:

[1334] The device uses the camera device to capture video in real time. This operation uses the camera API to capture video data frame by frame in JPEG or RAW format. The input is the video data obtained by the user launching the camera app and pointing the camera at a subject, and the output is the video frames converted into digital format.

[1335] Step 2:

[1336] Video transmission:

[1337] The device sends the captured video data to the server via a communication interface. During this process, the data is compressed and encrypted and sent via a REST API. The input is the captured video frame, and the output is the digital data received by the server.

[1338] Step 3:

[1339] Video Analysis:

[1340] The server analyzes the received video data using a deep learning model. Specifically, it uses TensorFlow and PyTorch to detect the subject's contours, position, light angle, color balance, etc. The input is the video data sent from the device, and the output is a set of parameters resulting from the analysis (e.g., subject position coordinates, light source angle, color balance, etc.).

[1341] Step 4:

[1342] Generate advice:

[1343] Based on the video analysis results, the server generates specific advice on professional shooting techniques, such as "move the subject a little more to the right" or "change the light source." The input is a set of parameters from the analysis results, and the output is text data of the generated advice.

[1344] Step 5:

[1345] Send Advice:

[1346] The server sends the generated advice to the device. The data is compressed and encrypted again and sent using the REST API. The input is the text data of the generated advice, and the output is the advice data received by the device.

[1347] Step 6:

[1348] Show Advice:

[1349] The terminal visually displays the advice received from the server on the user interface. Specifically, it conveys the advice to the user using messages and icons. The input is the advice data received from the server, and the output is a visual display that the user can see through the terminal.

[1350] Step 7:

[1351] Adjusting camera settings:

[1352] The device automatically adjusts camera settings such as exposure, focus, and white balance based on advice from the server, using the device's native API. The input is advice data from the server, and the output is the adjusted camera settings.

[1353] Step 8:

[1354] Shooting execution:

[1355] The user adjusts the camera's orientation and subject position according to the advice displayed on the device, and then presses the shutter button to take the photo. The input is the advice displayed on the device, and the output is the high-quality photo taken.

[1356] (Application example 1)

[1357] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1358] Adjusting object placement, light angle, and color balance using conventional camera devices and displays is difficult for users without specialized knowledge, often resulting in low-quality displays and placements. In particular, display placement that maximizes the appeal of products is important in brick-and-mortar stores, and optimal placement must be achieved efficiently and intuitively. To solve this problem, it is necessary to provide a system that allows anyone to easily perform high-quality displays and placements.

[1359] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1360] In this invention, the server includes means for receiving images from a camera device in real time, means for analyzing the received images and detecting object placement, light angle, and color balance, means for generating advice on photography techniques based on the detection results, means for displaying the generated advice, means for automatically adjusting camera device settings based on the advice, means for providing specific advice on object placement and lighting adjustment, and means for generating and displaying advice for optimizing product placement in a store. This enables anyone, even without specialized knowledge, to achieve professional-quality placement and display.

[1361] A "camera device" is a device for capturing video, and includes smartphones, digital cameras, and the like.

[1362] The "means for receiving video in real time" refers to a means for receiving video information transmitted from a camera device without delay and providing it for analysis.

[1363] The "means for detecting the position of an object, the angle of light, and the color balance" refers to a means for identifying the positional relationship of an object in an image, the angle of incidence of light, and the color arrangement state.

[1364] The "means for generating advice on photography techniques" is a means for generating specific advice showing how to take photographs to obtain optimal results, based on the detection results.

[1365] The "means for displaying the generated advice" refers to a means for visually presenting the generated advice to the user.

[1366] The "means for automatically adjusting camera device settings" refers to a means for automatically changing camera settings such as exposure, focus, and white balance based on the advice.

[1367] The "means for providing specific advice regarding the placement of objects and the adjustment of lighting" refers to a means for creating and providing advice regarding the optimization of the placement of objects and lighting in store displays.

[1368] The "means for generating and displaying advice for optimizing product placement in a store" is a means for creating advice for optimizing product placement in a physical store and visually presenting it to the user.

[1369] This invention is a system that uses a camera device to analyze images in real time, provides advice on object placement, lighting angle, and color balance, and automatically adjusts settings based on the results. This system is particularly applicable to optimizing product placement and displays in physical stores.

[1370] The server analyzes the video sent from the smartphone or smart glasses in real time. A deep learning model (such as TensorFlow or PyTorch) is used for the analysis to detect the placement of objects, the angle of light, color balance, etc. Based on the detection results, specific advice is generated, such as "Move the product up 5 cm" or "Dim the lighting a little." This generated advice is sent to the smartphone or smart glasses.

[1371] The device (smartphone or smart glasses) displays the advice received from the server on the user interface. For example, in the case of smart glasses, the advice is projected visually as icons or messages, which the user can use to adjust the display or product placement. The device also captures video data and sends it to the server. It also has the function of automatically adjusting camera settings (exposure, focus, white balance, etc.) based on the advice.

[1372] Users can adjust the placement of objects and lighting according to the advice displayed on their devices. For example, they can rearrange displays in a physical store to maximize the appeal of their products. In this way, professional-quality placement and display can be achieved without specialized knowledge.

[1373] Examples:

[1374] 1. The user wears the smart glasses and transmits the image of the display area to the server.

[1375] 2. The server analyzes the video and generates specific advice such as "Move the display up 5cm."

[1376] 3. The smart glasses provide visual advice to the user, who then adjusts the display accordingly.

[1377] Example prompt sentence:

[1378] "Make sure your product is centered and symmetrical."

[1379] "By dimming the lighting in this area a little, the merchandise will stand out."

[1380] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1381] Step 1:

[1382] The server receives video sent from the terminal in real time. Here, the input is video data from the terminal, and the output is the received video data. Specifically, this is a process in which video captured by the terminal's camera is sent to the server via the Internet.

[1383] Step 2:

[1384] The server analyzes the received video and detects the object placement, light angle, and color balance. The input is the video data received in step 1, and the output is data on the object placement, light angle, and color balance. This process uses a deep learning model (e.g., TensorFlow or PyTorch) to obtain the analysis results.

[1385] Step 3:

[1386] The server generates advice on photography techniques based on the analysis results. The input is the analysis data from step 2, and the output is specific advice information. For example, this process generates advice such as "Move the product up 5 cm" or "Dim the lighting a little."

[1387] Step 4:

[1388] The server sends the generated advice to the terminal. The input is the advice information generated in step 3, and the output is the advice information sent to the terminal. Specifically, this is a process of sending the generated text or icon information to the terminal via the network.

[1389] Step 5:

[1390] The device displays the received advice on the user interface. The input is the advice information received in step 4, and the output is a visual presentation to the user. For example, in the case of smart glasses, this is a process of projecting an icon or message into the user's field of view.

[1391] Step 6:

[1392] The device automatically adjusts the camera settings based on the advice. The input is the advice information received in step 4, and the output is the adjusted camera settings. For example, this is a process of changing exposure, focus, and white balance parameters in real time.

[1393] Step 7:

[1394] The user adjusts the placement of objects and lighting according to the advice displayed on the terminal. The input is the advice information displayed in step 5, and the output is the adjusted placement of objects and lighting. Specifically, this is a process in which the user manually changes the position of products and lighting settings.

[1395] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1396] This invention combines a system that analyzes video in real time using a camera device, provides photography advice, and automatically adjusts settings based on the results with an emotion engine that recognizes the user's emotions. This system analyzes video captured by a camera device (e.g., a smartphone camera) to detect the subject's position, lighting angle, and color balance. Based on the results, it generates advice on professional photography techniques and displays it to the user. Furthermore, the emotion engine recognizes the user's emotional state in real time and adjusts the content and display method of the advice depending on that emotional state. This allows anyone to easily take high-quality photos.

[1397] System Configuration

[1398] 1. Server

[1399] The server analyzes the video received from the camera in real time and generates advice based on the results. It also uses an emotion engine to recognize the user's emotional state and adjusts the advice based on the results. Specifically, it uses a deep learning model to perform highly accurate image analysis, detecting the subject's contours, position, light angle, and color balance. The server then sends the generated advice to devices such as smartphones.

[1400] 2. Terminal

[1401] The device is a smartphone equipped with a camera, and it displays advice from the server to the user and automatically adjusts settings. The device also has the function of sending images captured by the camera to the server. The device also displays the advice received from the server on the user interface, visually conveying it as icons and messages so that the user can intuitively understand it.

[1402] 3. Users

[1403] The user operates the camera and takes pictures according to the system's advice. By adjusting the camera's orientation and subject position based on the icons and messages displayed on the user interface, the user can take high-quality photos. The emotion engine also detects the user's emotional state and provides feedback accordingly.

[1404] Program processing overview (natural language explanation)

[1405] server:

[1406] The server analyzes the video frames received from the device. The video is analyzed frame by frame to detect the contours and position of the subject, the angle of light, and color balance.

[1407] Based on the detection results, it generates specific advice that can be useful when taking photos, such as "move the subject a little more to the right" or "change the light source."

[1408] The emotion engine analyzes the user's emotional state and tailors the advice accordingly: if the user is relaxed, it provides detailed advice, and if the user is tense, it provides simplified advice.

[1409] The generated advice is sent to the device.

[1410] Device:

[1411] The device displays the advice received from the server on the user interface, with specific instructions and icons displayed on the screen to help the user intuitively understand.

[1412] The device adjusts how it displays advice based on the analysis results of the emotion engine: if the user is relaxed, it displays detailed information, and if the user is nervous, it displays simplified information.

[1413] It automatically adjusts camera settings based on advice from the server, including exposure, focus, and white balance. For example, it compensates for exposure if backlighting is detected.

[1414] User:

[1415] The user adjusts the subject and camera position based on the advice displayed on the device.

[1416] Once you've followed the advice and optimized your composition and settings, simply press the shutter button to take the photo, resulting in professional-quality photos with ease.

[1417] After the photo is taken, the server provides additional feedback to help the user further improve their skills.

[1418] Specific examples

[1419] As an example, consider a case where a user takes a portrait photo with a sunset as the background.

[1420] server:

[1421] Detects the position of the setting sun and people, and recognizes backlight conditions.

[1422] It generates advice such as, "Try to change the camera angle a little so the sun is above your subject's head."

[1423] The emotion engine analyzes the user's facial expressions and recognizes that the user is relaxed. In this case, it provides detailed advice.

[1424] The generated advice is sent to the device.

[1425] Device:

[1426] The user interface provides detailed advice such as "Adjust your camera angle" and displays an arrow icon indicating the ideal camera angle.

[1427] Adjust the camera's exposure and change the settings so that the subject is not too dark.

[1428] User:

[1429] Follow the advice displayed on your device to adjust the camera angle and correct the subject's position.

[1430] Press the shutter button to take the picture. As a result, you can take a beautiful photo of the subject, taking advantage of the backlight.

[1431] In this way, the system of the present invention allows users to take high-quality photos simply and intuitively, and further optimizes the photography experience by providing feedback according to the user's emotional state.

[1432] The processing flow will be explained below.

[1433] Step 1:

[1434] The user launches the camera app on their smartphone, points the camera at the subject they want to photograph, and frames the subject on the screen.

[1435] Step 2:

[1436] The device captures video frames from the camera in real time, and the captured video frames are sent to the server at regular intervals.

[1437] Step 3:

[1438] The server analyzes video frames received from the device and uses deep learning models to detect the subject's contours, lighting angle, and color balance, identifying situations where the subject is too close to the center of the screen or where there is strong backlighting.

[1439] Step 4:

[1440] The server generates advice based on the analysis results. For example, if the subject is too close to the center, it generates specific advice such as "move the subject a little more to the right." The server then sends this advice to the device.

[1441] Step 5:

[1442] The device displays the advice received from the server on the user interface. Specific instructions and icons are displayed on the screen so that the user can understand intuitively. For example, the text "Adjust the camera angle" and an arrow icon are displayed.

[1443] Step 6:

[1444] The device automatically adjusts camera settings. For example, if backlighting is detected, it will correct the exposure and change settings to prevent the subject from appearing too dark. It also adjusts the content and display of advice based on the user's emotional state.

[1445] Step 7:

[1446] The server uses an emotion engine to analyze the user's emotional state, analyzing their facial expressions and voice in real time to determine whether they are relaxed or tense.

[1447] Step 8:

[1448] The server then adjusts the advice based on the emotion engine's analysis: if the subject is relaxed, it provides detailed advice, such as "change the direction of the light to reduce shadows," while if the subject is nervous, it provides simplified advice, such as "move the subject a little more to the right."

[1449] Step 9:

[1450] The device displays the adjusted advice received from the server on the user interface, and the user can refer to it to adjust the position of the subject and camera.

[1451] Step 10:

[1452] The user adjusts the subject and camera position according to the advice displayed on the device, and once the adjustments are complete, presses the shutter button to take the photo.

[1453] Step 11:

[1454] Your device will capture and store the image, and your settings and recommendations will help you capture the perfect photo.

[1455] Step 12:

[1456] If necessary, the server will analyze the image after shooting and provide additional feedback, which will be presented to the user as advice for future photography.

[1457] In this way, all steps work together to allow users to easily take professional-looking photos and receive feedback that adapts to their emotional state.

[1458] Example 2

[1459] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1460] Conventional camera systems require users to adjust the composition and settings themselves, requiring the skills and knowledge to take high-quality photos. Furthermore, they are unable to provide appropriate photography advice based on the user's emotional state, resulting in a suboptimal photography experience.

[1461] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1462] In this invention, the server includes means for receiving video from a camera device in real time, means for analyzing the received video and detecting the subject's position, light angle, and color balance, means for generating advice on photography techniques based on the detection results, means for analyzing the user's emotional state and adjusting the content and display method of the advice, means for displaying the generated advice, and means for automatically adjusting the camera device settings based on the advice. This allows users to optimize the photography process through an intuitive interface, making it easy to take high-quality photos. Furthermore, providing feedback according to the user's emotional state improves the photography experience.

[1463] A "camera device" is a device for capturing video, and generally refers to a digital camera, a video camera, a camera mounted on a smartphone, etc.

[1464] "Means for receiving video in real time" refers to a system and protocol for receiving video frames sent from a camera device in real time.

[1465] "Means for analyzing received video" refers to software and hardware that uses image processing technology and machine learning models to detect the subject's position, light angle, color balance, etc. from the received video data.

[1466] "Subject placement" refers to information indicating the position of the subject within the captured video.

[1467] "Light angle" refers to information indicating the angle at which the light source is irradiating the subject in the captured image.

[1468] "Color balance" refers to the distribution and harmony of colors within an image, and in particular the degree to which colors are accurately reproduced under white light.

[1469] "Advice on photography techniques" refers to specific instructions and suggestions that users need to take higher quality photos and videos, based on analyzed video data.

[1470] The "means for displaying the generated advice" refers to a device or interface that allows the user to visually understand the advice, and generally refers to a display or the screen of a mobile device.

[1471] "Means for automatically adjusting camera device settings" refers to software and hardware that allows the system to automatically change settings such as exposure, focus, and white balance of the camera based on the generated advice.

[1472] "Means for analyzing the user's emotional state" refers to software and hardware for analyzing the user's facial expressions, vocal tone, etc., and determining whether the user is relaxed, tense, or otherwise emotional.

[1473] "Means for adjusting the content and display of advice" refers to software and hardware for changing the level of detail and display format of the advice provided based on the user's emotional state.

[1474] "Feedback" refers to evaluations and suggestions for the next shoot provided to the user based on the results of the shoot.

[1475] This system uses a camera device to analyze video in real time, provide advice on shooting, and automatically adjust settings based on the results. It also incorporates an emotion engine that recognizes the user's emotions. This system operates through the interaction of a server, a terminal, and a user.

[1476] Server Roles and Operations

[1477] 1. Receiving video

[1478] The server receives video frames sent from the terminal in real time. To achieve real-time processing, video data is received at a fixed frame rate (e.g., 30 frames per second). It is desirable to use high-performance server hardware.

[1479] 2. Video Analysis

[1480] The server analyzes the received video frames. This analysis uses a deep learning model. Specifically, it applies an image analysis model such as "DeepVision" to detect the subject's contours, position, light angle, and color balance. Specifically, it analyzes the brightness and color components of each pixel in the frame and applies a filter that emphasizes the subject's edges.

[1481] 3. Advice Generation

[1482] Based on the detection results, the system generates specific advice for photography, such as "Moving the subject 20 pixels to the right will improve the composition," and sends this advice in text format to the device.

[1483] 4. Emotion analysis

[1484] The server analyzes the user's facial expression data using emotion recognition software such as "Emotion API." If the user is relaxed, detailed advice is generated, and if the user is tense, simplified advice is generated. Emotion analysis is performed based on the pattern of changes in facial expressions; for example, if the smile detection rate exceeds 80%, the user is judged to be "relaxed."

[1485] 5. Submitting Advice

[1486] The generated advice and how to display it based on emotions are sent to the device in JSON format.

[1487] Terminal roles and processing

[1488] 1. Acquiring footage

[1489] The device acquires video frames from the camera device, and the video is sent to the server in real time. Specifically, the device captures video using the smartphone's camera API and sends it to the server via a streaming protocol.

[1490] 2. Receiving Advice

[1491] The device receives the advice data sent from the server and parses it in JSON format.

[1492] 3. Displaying Advice

[1493] The device displays advice in the user interface, using a UI framework such as "One UI" to display specific instructions and icons on the screen so that users can understand intuitively.

[1494] 4. Automatically adjust camera settings

[1495] The device automatically adjusts the camera settings based on advice from the server. For example, if backlighting is detected, the device uses the exposure compensation function to increase brightness. Specifically, the device uses the camera API to call a function that adjusts the exposure level.

[1496] User Roles and Actions

[1497] 1. Check the advice

[1498] The user checks the advice displayed on the device. The advice is displayed as text and icons, so the content can be understood instantly.

[1499] 2. Camera Controls

[1500] The user adjusts the camera angle and the position of the subject according to the advice provided. For example, the user changes the position of the subject according to the advice "move the subject to the right."

[1501] 3. Shooting

[1502] Once the composition and settings are optimized, the user presses the shutter button to take the photo, resulting in a high-quality photo that reflects the system's advice.

[1503] 4. Review feedback

[1504] After taking a photo, the server provides additional feedback to identify areas for improvement for the next photo. The feedback includes a user's rating of the photo and new advice for the next photo.

[1505] Specific examples

[1506] If a user wants to take a portrait photo with a sunset in the background:

[1507] server:

[1508] The server uses image analysis models to detect the location of the setting sun and people, and recognize backlit conditions.

[1509] It generates advice such as, "Try to change the camera angle a little so the sun is above your subject's head."

[1510] Emotion recognition software analyzes the user's facial expressions and provides detailed advice if they are relaxed.

[1511] The generated advice is sent to the device.

[1512] Device:

[1513] It displays detailed advice such as "Adjust your camera angle" and displays an arrow icon indicating the ideal camera angle.

[1514] Adjust the camera's exposure and change the settings so that the subject is not too dark.

[1515] User:

[1516] Follow the advice displayed on your device to adjust the camera angle and correct the subject's position.

[1517] Press the shutter button to take the picture. As a result, you can take a beautiful photo of the subject, taking advantage of the backlight.

[1518] Prompt Sentence Examples

[1519] "I'm trying to take a portrait with the sunset in the background. I'd like some advice on how to position the camera, taking into account the subject and the angle of the light."

[1520] In this way, the system of the present invention allows users to take high-quality photos simply and intuitively, and further optimizes the photography experience by providing feedback that is responsive to the user's emotional state.

[1521] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1522] Step 1: Acquire footage

[1523] The device acquires video frames from the camera device and transmits them to the server in real time. Specifically, it uses the smartphone's camera API to capture video and transmits the video frames to the server at a constant frame rate (30 frames per second) via a streaming protocol. The input is the video frames acquired from the camera device, and the output is the transmission of real-time video data to the server.

[1524] Step 2: Receiving the video

[1525] The server receives video frames sent from the terminal. Specifically, the server uses a high-performance network interface to receive data in real time. The input is the video frame sent from the terminal, and the output is the video data stored in the server.

[1526] Step 3: Video analysis

[1527] The server analyzes the received video frames. Specifically, it uses the deep learning model "DeepVision" to analyze the brightness and color components of each frame and detect the subject's contours, position, light angle, and color balance. The input is the received video data, and the output is information on the detected subject's position, light angle, and color balance.

[1528] Step 4: Advice Generation

[1529] The server generates specific advice about shooting based on the results of video analysis. Specifically, it generates a text message based on the image analysis results, instructing the user on how to adjust the camera and subject. Specific advice such as "Moving the subject 20 pixels to the right will improve the composition" is generated and output in JSON format. The input is the video analysis results, and the output is the generated advice data.

[1530] Step 5: Sentiment Analysis

[1531] The server analyzes the user's facial expression data. Specifically, it uses emotion recognition software "Emotion API" to analyze the user's emotional state from their facial expressions. It determines whether the user is relaxed or tense, and adjusts the content and display format of the advice accordingly. The input is the user's facial expression data, and the output is the analyzed information on the user's emotional state.

[1532] Step 6: Submitting Advice

[1533] The server sends the generated advice and the emotion analysis results to the terminal. Specifically, it sends the generated advice and the emotion-based display method together in JSON format to the terminal. The input is the generated advice data and the emotion analysis results, and the output is JSON data containing them.

[1534] Step 7: Viewing Advice

[1535] The device analyzes the advice data received from the server and displays it on the user interface. Specifically, it uses "One UI" to display a text message such as "Move the subject to the right" or an arrow icon. The input is the advice data received from the server, and the output is the advice displayed on the user interface.

[1536] Step 8: Automatically adjust camera settings

[1537] The device automatically adjusts the camera settings based on advice from the server. Specifically, it uses the camera API to increase brightness using the exposure compensation function when backlighting is detected, for example. The input is the advice data received from the server, and the output is the automatically adjusted camera settings.

[1538] Step 9: Review the advice

[1539] The user checks the advice displayed on the device and prepares to take the photo. Specific actions include checking the specific instructions and icons displayed on the device screen. The input is the advice displayed on the device, and the output is the user's understanding and preparation.

[1540] Step 10: Shoot

[1541] The user adjusts the camera settings and subject position according to the device's advice, then presses the shutter button to take the photo. Specifically, the user follows the instructions displayed on the device, adjusts the camera angle and subject position, and finally presses the shutter button. The input is the advice displayed on the device and the user's actions, and the output is the high-quality photo taken.

[1542] Step 11: Review feedback

[1543] After taking a photo, the user checks the feedback sent from the server. Specifically, the user checks the evaluation based on the shooting results and suggestions for the next shoot. The input is the feedback data sent from the server after the photo is taken, and the output is the user's understanding and skill improvement for the next shoot.

[1544] (Application example 2)

[1545] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1546] Conventional camera systems require users to have a high level of specialized knowledge to take high-quality photos, and the results are often influenced by the user's emotional state. Therefore, there is a need for a system that allows even general users without specialized knowledge to take high-quality photos intuitively and easily. Furthermore, users who create advertising content, such as influencers, require particularly accurate advice and emotional feedback. Conventional systems do not adequately meet these requirements, and a more user-friendly and advanced system is needed.

[1547] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving video from a camera device in real time, means for analyzing the received video and detecting the subject's position, light angle, and color balance, and means for generating advice on photography techniques based on the detection results. This enables means for displaying the generated advice, means for automatically adjusting the camera device settings based on the advice, means for analyzing the user's emotions and adjusting the advice content based on the user's emotional state, means for displaying specific instructions for photography, and means for providing professional photography advice that optimizes the user's photography experience.

[1548] A "camera device" is a device for receiving and capturing video in real time.

[1549] The "means for analyzing received images" is a technology for detecting the subject's position, light angle, and color balance based on the image data received from the camera device.

[1550] The "means for generating advice" is a method for generating specific instructions or suggestions to assist with photography based on the analysis results.

[1551] The "means for displaying generated advice" is a technique for visually displaying advice as icons or messages so that the user can intuitively understand it.

[1552] The "means for automatically adjusting settings" is a technology for automatically adjusting settings such as exposure, focus, and white balance of a camera based on the generated advice.

[1553] "Means for analyzing user emotions" refers to technology that analyzes the user's emotional state during shooting and provides appropriate feedback based on those emotions.

[1554] The "means for adjusting the content of advice based on the emotional state" is a technique for adjusting the details and display method of advice according to the analyzed emotional state of the user.

[1555] The "means for displaying specific instructions" is a technique for displaying specific operational instructions for shooting in a visually easy-to-understand manner to the user.

[1556] "Means for optimizing the user's photography experience" refers to methods for adjusting various settings and feedback so that the user can take photos in the best overall condition.

[1557] The "means for providing professional photography advice" is a technology for providing appropriate advice to users in real time based on specialized knowledge.

[1558] This invention realizes an influencer photography support application for advertising purposes on a smartphone. A specific embodiment of this system is described below.

[1559] server

[1560] The server has the following features:

[1561] 1. Video reception and analysis:

[1562] It receives video from the camera in real time, and analyzes the video frame by frame to detect subject placement, lighting angle, and color balance using a deep learning framework (e.g., TensorFlow or PyTorch).

[1563] 2. Generating Advice:

[1564] Based on the detection results, it generates specific advice on photography techniques, such as "move the subject a little more to the right" or "change the light source."

[1565] 3. User sentiment analysis:

[1566] It uses an emotion analysis engine (e.g., Emotion API) to recognize the user's emotional state and provide appropriate advice based on the results: detailed advice if the user is relaxed, simple advice if the user is tense.

[1567] 4. Submitting Advice:

[1568] The generated advice is sent to the smartphone device, including feedback based on the user's emotional state.

[1569] Terminal

[1570] The terminal has the following features:

[1571] 1. Video capture and transmission:

[1572] It uses the camera API to capture video and send it to the server in real time.

[1573] 2. Display Advice:

[1574] The advice received from the server is displayed in the user interface, with specific instructions and icons displayed so that the user can intuitively understand.

[1575] 3. Automatically adjust camera settings:

[1576] Automatically adjusts camera settings (exposure, focus, white balance, etc.) based on advice from the server, for example correcting exposure if backlighting is detected.

[1577] User

[1578] The user actions are as follows:

[1579] 1. Shooting according to the advice:

[1580] Follow the advice displayed on your device to adjust the subject and camera position.

[1581] Once you have determined the optimal composition and settings, press the shutter button to take the picture.

[1582] 2. Use feedback:

[1583] After shooting, you can use additional feedback from the server to improve your photography skills.

[1584] Specific examples

[1585] As a concrete example of this system, consider a situation in which a user wants to photograph a new cosmetic product with a sunset as the background. The server analyzes the sunset and the subject's position, recognizes the backlighting, and generates advice such as, "Change the camera angle slightly so that the sun is above the subject's head." If the emotion analysis engine analyzes the user's facial expression and detects a relaxed state, it provides detailed advice. This advice is sent to the device, which displays an arrow icon indicating the ideal angle. The user adjusts the camera angle accordingly and presses the shutter button to take the photo. The result is a professional-quality photo with a beautiful sunset as the background.

[1586] Prompt Sentence Examples

[1587] "To photograph cosmetics against a sunset background, adjust the camera angle and center the subject. Then adjust the lighting to highlight the cosmetics."

[1588] As a result, the present invention provides a system that allows anyone to easily take high-quality advertising photos by providing feedback according to the user's emotional state.

[1589] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1590] Step 1:

[1591] Video acquisition and transmission

[1592] Subject: Terminal

[1593] Specific operation: Captures video in real time through the device's camera API, and sends the video frames captured by the camera to the server at regular intervals.

[1594] Input: Raw video data from the camera.

[1595] Output: Video frames sent to the server in real time.

[1596] Step 2:

[1597] Video reception and analysis

[1598] Subject: Server

[1599] What it does: Receives video sent from the device. Uses a deep learning framework to analyze video frames in real time, specifically detecting subject placement, lighting angle, and color balance.

[1600] Input: Video frames sent from the device.

[1601] Output: Information about subject placement, information about light angle, information about color balance.

[1602] Step 3:

[1603] Generating Advice

[1604] Subject: Server

[1605] Specific actions: Based on the data obtained from video analysis, specific advice on shooting techniques is generated, such as "move the subject to the center" or "change the light source."

[1606] Input: Information about subject placement, light angle, and color balance.

[1607] Output: Specific advice on photography techniques.

[1608] Step 4:

[1609] User sentiment analysis

[1610] Subject: Server

[1611] Specific operation: Using the Emotion API, etc., the system analyzes the user's emotional state from the facial expressions captured by the camera. The system then adjusts the advice accordingly. For example, if the user is relaxed, the system generates detailed advice, and if the user is nervous, the system generates simple advice.

[1612] Input: A video frame containing the user's facial expression.

[1613] Output: Appropriate feedback and advice depending on the user's emotional state.

[1614] Step 5:

[1615] Sending and viewing advice

[1616] Subject: Server, Terminal

[1617] Specific operation: The server sends the generated advice to the device. The device displays the received advice on the user interface, displaying specific instructions and icons that the user can intuitively understand.

[1618] Input: Generated advice,feedback based on emotional state.

[1619] Output: Specific shooting advice and feedback displayed in the user interface.

[1620] Step 6:

[1621] Automatic camera settings adjustment

[1622] Subject: Terminal

[1623] Specific behavior: Automatically adjusts camera settings (e.g., exposure, focus, white balance, etc.) based on advice received from the server. For example, correcting exposure when backlighting is detected.

[1624] Input: The generated advice.

[1625] Output: Camera settings automatically adjusted.

[1626] Step 7:

[1627] User operation and shooting

[1628] Subject: User

[1629] Specific operations: Follow the advice displayed on the device to adjust the position of the subject and camera, obtain the optimal composition and settings, and then press the shutter button to take the photo.

[1630] Input: The advice displayed on the terminal.

[1631] Output: Professional quality photos taken.

[1632] Step 8:

[1633] Post-shoot feedback

[1634] Subject: Server, User

[1635] Specific operation: The server analyzes the captured image and provides additional feedback to the user, who can use this feedback to improve their next capture.

[1636] Input: The photo taken.

[1637] Output: Additional feedback and advice.

[1638] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1639] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1640] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1641] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1642] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1643] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1644] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1645] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1646] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1647] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1648] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1649] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1650] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1651] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1652] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1653] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1654] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1655] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1656] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1657] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1658] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1659] The following is further disclosed regarding the above embodiment.

[1660] (Claim 1)

[1661] means for receiving images from a camera device in real time;

[1662] A means for analyzing the received video and detecting the subject's position, light angle, and color balance;

[1663] means for generating advice on photography techniques based on the detection results;

[1664] a means for displaying the generated advice;

[1665] A means to automatically adjust camera settings based on the advice

[1666] Including system.

[1667] (Claim 2)

[1668] 2. The system of claim 1, further comprising: means for visually displaying the generated advice to the user as a corresponding icon or message; and means for automatically adjusting exposure, focus, and white balance of the camera device based on the advice.

[1669] (Claim 3)

[1670] 3. The system of claim 1 or claim 2, further comprising means for providing post-capture editing feedback.

[1671] "Example 1"

[1672] (Claim 1)

[1673] means for receiving images from a camera device in real time;

[1674] A means of analyzing the received video using a deep learning model to detect the subject's position, light angle, and color balance;

[1675] a means for generating advice on professional photography techniques based on the detection results;

[1676] means for transmitting the generated advice to a terminal;

[1677] A means for automatically adjusting camera device settings based on advice displayed on the device

[1678] Including system.

[1679] (Claim 2)

[1680] 2. The system of claim 1, further comprising: means for visually displaying the generated advice to the user as a corresponding icon or message; and means for automatically adjusting exposure, focus, and white balance of the camera device based on the advice.

[1681] (Claim 3)

[1682] 10. The system of claim 1, further comprising means for providing post-capture editing feedback.

[1683] "Application Example 1"

[1684] (Claim 1)

[1685] means for receiving images from a camera device in real time;

[1686] A means for analyzing the received video and detecting the object placement, light angle, and color balance;

[1687] means for generating advice on photography techniques based on the detection results;

[1688] a means for displaying the generated advice;

[1689] means for automatically adjusting settings of the camera device based on the advice;

[1690] a means of providing specific advice on object placement and lighting adjustments;

[1691] A means of generating and displaying advice for optimizing in-store product placement

[1692] A system including:

[1693] (Claim 2)

[1694] 2. The system of claim 1, further comprising: means for visually displaying the generated advice to the user as a corresponding icon or message; and means for automatically adjusting exposure, focus, and white balance of the camera device based on the advice.

[1695] (Claim 3)

[1696] 10. The system of claim 1, further comprising means for providing post-capture editing feedback.

[1697] "Example 2: Combining Emotion Engines"

[1698] (Claim 1)

[1699] means for receiving images from a camera device in real time;

[1700] A means for analyzing the received video and detecting the subject's position, light angle, and color balance;

[1701] means for generating advice on photography techniques based on the detection results;

[1702] a means for displaying the generated advice;

[1703] means for automatically adjusting settings of the camera device based on the advice;

[1704] Analyzing the user's emotional state and adjusting the content and display of advice

[1705] Including system.

[1706] (Claim 2)

[1707] 2. The system of claim 1, further comprising: means for visually displaying the generated advice to the user as a corresponding icon or message; and means for automatically adjusting exposure, focus, and white balance of the camera device based on the advice.

[1708] (Claim 3)

[1709] 10. The system of claim 1, further comprising means for providing post-capture editing feedback.

[1710] "Application example 2 when combining emotion engines"

[1711] (Claim 1)

[1712] means for receiving images from a camera device in real time;

[1713] A means for analyzing the received video and detecting the subject's position, light angle, and color balance;

[1714] means for generating advice on photography techniques based on the detection results;

[1715] a means for displaying the generated advice;

[1716] means for automatically adjusting settings of the camera device based on the advice;

[1717] means for analyzing the user's emotions and adjusting the content of advice based on the user's emotional state;

[1718] means for displaying specific instructions for taking a photograph;

[1719] A means to provide professional photography advice to optimize users' photography experience

[1720] Including system.

[1721] (Claim 2)

[1722] 2. The system of claim 1, further comprising: means for visually displaying the generated advice to the user as a corresponding icon or message; and means for automatically adjusting exposure, focus, and white balance of the camera device based on the advice.

[1723] (Claim 3)

[1724] 10. The system of claim 1, further comprising means for providing post-capture editing feedback. [Explanation of symbols]

[1725] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving images from a camera device in real time; A means for analyzing the received video and detecting the subject's position, light angle, and color balance; means for generating advice on photography techniques based on the detection results; a means for displaying the generated advice; A means to automatically adjust camera settings based on the advice Including system.

2. 2. The system of claim 1, further comprising: means for visually displaying the generated advice to the user as a corresponding icon or message; and means for automatically adjusting exposure, focus, and white balance of the camera device based on the advice.

3. 3. The system of claim 1 or claim 2, further comprising means for providing post-shooting editing feedback.

Images