Method for generating digital frame content using emotion-based voice data and digital frame system

Abstract

A method for generating electronic photo frame content using emotion-based voice data according to one embodiment of the present application may include: receiving shooting data, activity log data, and diary data from a user terminal; analyzing emotions based on the diary data; extracting keywords based on the shooting data and the activity log data; generating natural language data based on the emotion analysis data and keywords; and generating voice data using the generated natural language data and transmitting the generated voice data to an electronic photo frame for output.

Description

Technology Field

[0001] The present application relates to a method for generating electronic picture frame content using emotion-based voice data and an electronic picture frame system. Background Technology

[0002] Recently, various technologies aimed at strengthening communication among family members are being developed. In particular, digital photo frames are widely used as a means to maintain emotional bonds among family members through photos and videos related to child-rearing.

[0003] However, conventional digital photo frames are primarily limited to the playback of static images or videos. Furthermore, conventional digital photo frames are limited to manual transmission of photos and videos or providing only simple playback functions. Consequently, conventional digital photo frames lack user interaction and the depth of emotion conveyed is limited.

[0004] In addition, while there are numerous apps based on parenting data, the technology to convert and utilize this data into emotional content is still lacking.

[0005] Meanwhile, with the advancement of sentiment analysis and speech synthesis technologies, various application services capable of extracting or expressing emotions based on text or voice data are emerging. These technologies can contribute to expressing or conveying users' emotions more richly.

[0006] Therefore, there is an increasing demand for technology that enables richer and more emotional family communication compared to existing methods by incorporating emotion-based voice data into digital photo frame content. Prior art literature

[0007] Published Patent Application 10-2007-0010894 (Published Jan. 24, 2007) The problem to be solved

[0008] The problem that the present invention aims to solve is to provide a method for generating electronic photo frame content using emotion-based voice data and an electronic photo frame system.

[0009] In addition, the objective of the present invention is to provide a method for generating electronic photo frame content and an electronic photo frame system that can automatically generate and provide electronic photo frame content formed in the voice of a specific person when shooting data, diary data, and activity log data are input, by utilizing an artificial intelligence model trained on the voice of a specific person.

[0010] The problems that the present invention aims to solve are not limited to those described above, and problems not mentioned will be clearly understood by those skilled in the art from this specification and the attached drawings. means of solving the problem

[0011] A method for generating electronic photo frame content using emotion-based voice data according to an embodiment of the present invention may include: receiving shooting data, activity log data, and diary data from a user terminal; analyzing emotions based on the diary data; extracting keywords based on the shooting data and the activity log data; generating natural language data based on the sentiment analysis data and keywords; and generating voice data using the generated natural language data and transmitting the generated voice data to an electronic photo frame for output.

[0012] In addition, an electronic photo frame system utilizing emotion-based voice data according to an embodiment of the present invention comprises: a communication unit; a processor; and a memory; wherein the processor may include: a data receiving unit that receives shooting data, activity log data, and diary data from a user terminal; an emotion analysis unit that analyzes emotions based on the diary data; a keyword extraction unit that extracts keywords based on the shooting data and the activity log data; a natural language data generation unit that generates natural language data based on the emotion analysis data and keywords; and a voice data generation unit that generates voice data using the generated natural language data.

[0013] The means for solving the problem of the present invention are not limited to the means for solving the problem described above, and unmentioned means for solving the problem will be clearly understood by those skilled in the art from this specification and the attached drawings. Effects of the invention

[0014] According to a method for generating electronic photo frame content using emotion-based voice data and an electronic photo frame system according to an embodiment of the present invention, voice data generated from a child's voice and captured data are displayed in an electronic photo frame, thereby inducing richer emotional interaction among family members.

[0015] In addition, according to one embodiment of the present invention, emotion-based customized electronic photo frame content can be automatically generated from input data without the need for the user to edit it separately.

[0016] In addition, according to one embodiment of the present invention, children, the elderly, and others who have difficulty reading text can intuitively understand information using only images and voice. Brief explanation of the drawing

[0017] FIG. 1 is a drawing illustrating the operation of an electronic picture frame system according to one embodiment of the present invention. FIG. 2 is a drawing illustrating the configuration of an electronic picture frame system according to one embodiment of the present invention. FIG. 3 is a drawing illustrating the detailed configuration of a processor according to one embodiment of the present invention. FIG. 4 is a diagram illustrating a method in which an electronic photo frame system according to an embodiment of the present invention generates and provides emotion-based electronic photo frame content based on shooting data, activity log data, and diary data. FIG. 5 is a drawing illustrating an example of a user terminal and an electronic picture frame displayed according to an embodiment of the present invention. Specific details for implementing the invention

[0018] The aforementioned objectives, features, and advantages of the present application will become more apparent from the following detailed description in conjunction with the accompanying drawings. However, as the present application is subject to various modifications and may have various embodiments, specific embodiments are illustrated in the drawings and described in detail below.

[0019] Throughout the specification, identical reference numbers generally represent identical components. Additionally, components with identical functions within the same scope of concept appearing in the drawings of each embodiment are described using the same reference numeral, and redundant descriptions thereof are omitted.

[0020] If it is determined that a detailed description of known functions or configurations related to this application could unnecessarily obscure the essence of this application, such detailed description is omitted. Furthermore, numbers used in the description of this specification (e.g., First, Second, etc.) are merely identifiers to distinguish one component from another.

[0021] Furthermore, the suffixes "module" and "part" for components used in the following embodiments are assigned or used interchangeably solely for the ease of drafting the specification, and do not inherently possess distinct meanings or roles.

[0022] In the following examples, singular expressions include plural expressions unless the context clearly indicates otherwise.

[0023] In the following embodiments, terms such as "include" or "have" mean that the features or components described in the specification are present, and do not preclude the possibility that one or more other features or components may be added.

[0024] In the drawings, the size of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are arbitrarily depicted for convenience of explanation, and the present invention is not necessarily limited to what is illustrated.

[0025] Where an embodiment can be implemented differently, the order of a particular process may be performed differently from the order described. For example, two processes described consecutively may be performed substantially simultaneously or proceed in the reverse order of the description.

[0026] In the following embodiments, when components are described as being connected, the case includes not only instances where the components are directly connected but also instances where components are indirectly connected by interposing them in between.

[0027] For example, when it is stated in this specification that components, etc. are electrically connected, it includes not only cases where the components, etc. are directly electrically connected, but also cases where components, etc. are interposed in between and are indirectly electrically connected.

[0028] Hereinafter, with reference to FIGS. 1 to 5, a method for generating electronic photo frame content using emotion-based voice data according to the present invention and an electronic photo frame system for performing the same will be described.

[0029] FIG. 1 is a drawing illustrating the operation of an electronic picture frame system according to one embodiment of the present invention.

[0030] As illustrated in FIG. 1, the electronic photo frame system (100) can receive shooting data, activity log data, and diary data from a user terminal (10). For example, the user terminal (10) may include a smartphone, tablet, laptop, and desktop.

[0031] The user terminal (10) may include a camera (not shown) and a microphone (not shown) to generate shooting data including images, videos, and voice. The shooting data may include time stamp data at the time of shooting.

[0032] Additionally, the user terminal (10) may receive text data. For example, the text data may include activity log data and diary data.

[0033] More specifically, the user terminal (10) may receive activity log data at the time of shooting. The activity log data is structured data in which a guardian records the daily care activities of an infant in chronological order, and may include data related to childcare, such as the child's physiological / behavioral (type of activity, state, and daily routine, etc.).

[0034] Additionally, the user terminal (10) may input diary data at the time of shooting. The diary data may include data in which a guardian (or parent) emotionally / narratively records the growth process and daily moments of the infant. The diary data may have the characteristics of unstructured text data rather than a structured form, and may include the guardian's subjective interpretation and emotional response.

[0035] The electronic photo frame system (100) can generate emotion-based voice data by utilizing shooting data, activity log data, and diary data received from the user terminal (10).

[0036] More specifically, the electronic picture frame system (100) can output emotion analysis data by processing diary data using natural language processing. The electronic picture frame system (100) can classify diary data by emotion based on an emotion classification model. The electronic picture frame system (100) can output the classified emotional states as emotion analysis data. For example, the emotion analysis data may include emotional states such as joy, anger, sadness, anxiety, hurt, embarrassment, worry, and interest.

[0037] Additionally, the electronic photo frame system (100) can extract keywords from activity log data and shooting data. For example, the electronic photo frame system (100) can classify keywords by processing the activity log data in natural language. The electronic photo frame system (100) can extract activity keywords based on the activity log data at the time the shooting data was captured. Additionally, the electronic photo frame system (100) can extract image keywords by recognizing objects through image analysis of the shooting data. Additionally, the electronic photo frame system (100) can determine key keywords based on the extracted activity keywords and image keywords.

[0038] Additionally, the electronic picture frame system (100) can generate natural language data based on sentiment analysis data and keywords. For example, the electronic picture frame system (100) can input sentiment analysis data and keywords into a natural language generation model. The electronic picture frame system (100) can structure the sentiment analysis data and keywords to generate sentiment-based narration as natural language data.

[0039] Not limited thereto, the electronic picture frame system (100) may further include time stamp data at the time the shooting data was taken and spatial data of the shooting data. By utilizing spatial and temporal information, the electronic picture frame system (100) creates a flow of story rather than simple data, thereby enhancing the context.

[0040] The electronic picture frame system (100) can generate voice data by utilizing natural language data and transmit the generated voice data to the electronic picture frame (20) for output. For example, the electronic picture frame system (100) can generate voice data by using natural language data as input to a voice generation model. The voice generation model can be trained to collect speech voice samples, extract features of the collected speech voice samples into vectors, and synthesize them to output voice data.

[0041] That is, the voice generation model can generate voice data from input natural language data as the speech of a specific person trained on it. In addition, the voice generation model can also adjust the pitch and speed of the voice data based on the sentiment of the input natural language data. Thus, the electronic picture frame system (100) can generate and provide voice data as content to which an emotional narrative is assigned.

[0042] Here, the electronic photo frame (20) can output the received voice data together with the captured data. The electronic photo frame (20) can display the captured data on a screen and output voice data through a speaker. The electronic photo frame (20) can output the emotion-based voice data generated by the electronic photo frame system (100) as emotional content.

[0043] Additionally, the electronic photo frame (20) may request and receive emotional content from the electronic photo frame system (100). More specifically, the electronic photo frame (20) may display a screen containing icons that allow checking information about shooting data and voice data. For example, if there is no voice data corresponding to specific shooting data, the electronic photo frame (20) may request the electronic photo frame system (100) to generate voice data for specific shooting data in response to input from a user who selects an icon requesting voice data.

[0044] At this time, the electronic photo frame system (100) can generate emotion-based voice data for the corresponding photo data after checking the photo data, activity log data, and diary data. Not limited thereto, if the electronic photo frame system (100) confirms that the activity log data and diary data for the requested photo data have not been entered, it may request the user terminal (10) to enter the activity log data and diary data.

[0045] FIG. 2 is a diagram illustrating the configuration of an electronic picture frame system according to an embodiment of the present invention. Referring to FIG. 2, the electronic picture frame system (100) may include a communication unit (110), a processor (120), and a memory (130).

[0046] The communication unit (110) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic picture frame system (100) and an external electronic device (user terminal (10), electronic picture frame (20)) and the performance of communication through the established communication channel.

[0047] The communication unit (110) operates independently of the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication unit (110) may include a wireless communication module (e.g., cellular communication module, short-range wireless communication module, GPRS (General Packet Radio Service), Wi-Fi, Bluetooth, 5G, LTE, or GNSS (global navigation satellite system) communication module, etc.) or a wired communication module (e.g., LAN (local area network) communication module, or power line communication module). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips).

[0048] The processor (120) can execute software to control at least one other component (e.g., hardware or software component) of the electronic picture frame system (100) connected to the processor (120) and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from another component (e.g., communication unit (110)) in volatile memory, process the commands or data stored in volatile memory, and store the resulting data in non-volatile memory. According to one embodiment, the processor (120) may include a main processor (e.g., central processing unit or application processor) or an auxiliary processor (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor) that can be operated independently or together with it.

[0049] According to one embodiment, an auxiliary processor (e.g., a neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, within the electronic picture frame system (100) itself where the artificial intelligence is performed, or through a separate server. The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers. An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0050] The memory (130) can store various data used by at least one component (e.g., processor (120)) of the electronic picture frame system (100). The data may include, for example, input data or output data for software and related commands. The memory (130) may include volatile memory or non-volatile memory.

[0051] FIG. 3 is a diagram illustrating the detailed configuration of a processor according to an embodiment of the present invention. As shown in FIG. 3, the processor (120) may include a data receiving unit (121), an emotion analysis unit (122), a keyword extraction unit (123), a natural language data generation unit (124), and a voice data generation unit (125).

[0052] The data receiving unit (121) can receive shooting data, activity log data, and diary data. For example, the shooting data may include images, videos, and audio. More specifically, the shooting data may include all data recorded in video or photos, such as the child's growth, behavior, facial expressions, and activities related to childcare. The shooting data may include time stamp data at the time of shooting. The time stamp data is time information indicating the time when the shooting data occurred, and may include the date and time.

[0053] In addition, activity log data is structured data in which a caregiver records the daily care activities of an infant or toddler in chronological order, and may include data related to child-rearing, such as the child's physiological and behavioral aspects (type of activity, status, routine, etc.). For example, activity log data is structured data that records the daily activities of an infant or toddler (e.g., feeding, sleeping, bowel movements, bathing, crying, playing, etc.) in chronological order, and may include the time, type of activity, details, and observed reactions. Activity log data may be received in text format.

[0054] Diary data may include records made by guardians (or parents) in an emotional or descriptive manner, documenting the growth process and daily moments of infants and toddlers. Diary data may possess the characteristics of unstructured text data rather than a structured form, and may include the guardian's subjective interpretations and emotional responses. More specifically, diary data may be written about the child's emotions, thoughts, experiences, and activities observed from the guardian's perspective, using emotional expressions and subjective interpretations.

[0055] The sentiment analysis unit (122) can generate sentiment analysis data based on an artificial intelligence model. For example, the sentiment analysis unit (122) may include a sentiment analysis model. The sentiment analysis model can take diary data as input and output sentiment analysis data.

[0056] Diary data can be preprocessed using a more specific sentiment analysis unit (122) natural language processing (NLP) model. Natural language processing data can be generated based on the diary data.

[0057] For example, the sentiment analysis unit (122) can perform morphological analysis, stop word removal, and sentence separation through a natural language processing model. The natural language processing model can convert the diary data into noise-free text by removing unnecessary symbols, emoticons, special characters, and duplicate spaces. Additionally, the natural language processing model can structure the converted text by dividing it into sentence units or word units (tokens). Additionally, the natural language processing model can identify the semantic structure by tagging each word to determine what part of speech it is, such as a noun, verb, or adjective. Furthermore, the natural language processing model can increase the focus of the sentiment classification model by removing stop words, such as particles, conjunctions, and interjections, that are unnecessary for semantic analysis.

[0058] The sentiment analysis unit (122) can input natural language processing data into the sentiment analysis model. The sentiment analysis model can classify the input diary data by emotion. For example, the sentiment analysis model can output sentiment analysis data including emotional states such as joy, anger, sadness, anxiety, hurt, embarrassment, worry, and interest.

[0059] Here, the sentiment analysis model can output multiple sentiment states by selecting the class with the maximum probability of the sentiment state, or by selecting based on at least one probability value without being limited thereto. In this case, the sentiment analysis model can predict sentiment states using a multi-class classification method.

[0060] For example, the sentiment analysis model can output {“1st sentiment” : 0.78, “2nd sentiment” : 0.61}. Therefore, since the sentiment analysis model can output probability values ​​for multiple sentiments from diary data, it is possible to apply multiple sentiments.

[0061] Sentiment analysis models may include deep learning-based KoBERT, KLUE-BERT, or custom fine-tuned sentiment classification models.

[0062] The keyword extraction unit (123) can extract keywords from activity log data and shooting data. For example, the keyword extraction unit (123) can extract and classify keywords by processing activity log data and shooting data in natural language. More specifically, the keyword extraction unit (123) can extract activity keywords by processing activity log data at the time the shooting data was captured in natural language. The keyword extraction unit (123) can select the most important keyword by comparing the semantic similarity between a sentence and a keyword candidate, summarize a list of important keywords, or extract activity keywords based on pre-set core keywords.

[0063] Additionally, the keyword extraction unit (123) can extract image keywords by recognizing objects through image analysis of the captured data. For example, the keyword extraction unit (123) can recognize objects in the images of the captured data based on YOLO (You Only Look Once) or MobileNet. The keyword extraction unit (123) can detect the object location (Bounding Box) and object type (Label) within the image. The keyword extraction unit (123) can extract the object type (Label) as a keyword. Additionally, the keyword extraction unit (123) can filter only those with high reliability among the detected object types, or determine objects that appear frequently as important keywords.

[0064] Thus, the keyword extraction unit (123) can determine the main keywords based on the extracted activity keywords and image keywords.

[0065] The natural language data generation unit (124) can generate natural language data based on sentiment analysis data and keywords. Not limited thereto, the natural language data generation unit (124) may further include time stamp data at the time the shooting data was taken and spatial data of the shooting data.

[0066] For example, the natural language data generation unit (124) may include a natural language generation model. The natural language generation model may include a GPT family (e.g., KoGPT, GPT-2 fine-tuned, T5) or an LLM-based custom NLG model. More specifically, the natural language data generation unit (124) may output natural language data in the form of narration based on the natural language generation model. The natural language generation model may generate narration according to a preset prompt. The preset prompt may store the narration style, tone, length, etc.

[0067] Here, the natural language generation model can generate emotion-based natural language data by structuring emotion analysis data and keywords. In addition, the natural language generation model can enhance the context of the natural language data by utilizing timestamp data and spatial data to include spatial and temporal information, thereby specifying the flow of the story and the space. Thus, the natural language data generation unit (124) can reinterpret data related to childcare into emotional content.

[0068] The voice data generation unit (125) can generate voice data based on natural language data. The voice data generation unit (125) can output voice data by using natural language data as input to a voice generation model. For example, the voice generation model can convert natural language data in text format into voice format using a Text To Speech (TTS) method. The voice generation model can be formed by a combination of SV2TTS / Resemblyzer, Tacotron2, and WaveGlow.

[0069] The voice data generation unit (125) can train a voice generation model to generate voice data from input natural language data as the utterance of a specific person. The voice data generation unit (125) can set the tone, utterance speed, sentence structure, prompt style, etc. based on the sentiment of the input natural language data. For example, if the input natural language data contains “tired,” the voice data generation unit can generate voice data such that the sentence has a low tone and a slow tempo.

[0070] Additionally, the output voice data can be set to be output in the spoken language of the voice model. More specifically, the voice data can be expressed in the first-person spoken language of a specific learned person by being spoken in the first-person spoken language of the voice model. Thus, the voice data generation unit (125) can generate and provide voice data with an emotional narrative as content.

[0071] The electronic photo frame (20, illustrated in FIG. 1) can receive and output voice data. The electronic photo frame (20) can not only output shooting data but also output voice data with an emotional narrative. Thus, the electronic photo frame (20) can provide multimodal content including text, emotions, voice, etc.

[0072] That is, the electronic picture frame system (100) can automatically generate and provide emotion-based customized electronic picture frame content from input data without the user needing to edit it separately. Therefore, children, the elderly, and others who have difficulty reading text can intuitively understand the information using only images and voice.

[0073] FIG. 4 is a diagram illustrating a method in which an electronic photo frame system according to an embodiment of the present invention generates and provides emotion-based electronic photo frame content based on shooting data, activity log data, and diary data.

[0074] Referring to FIG. 4, in step (S110), the electronic photo frame system (100) can receive shooting data, activity log data, and diary data from a user terminal (10, shown in FIG. 1). The electronic photo frame system (100) can receive shooting data including images, videos, and voice from the user terminal (10). Additionally, the electronic photo frame system (100) can receive activity log data and diary data in text format input into the user terminal (10).

[0075] Here, activity log data may include activity data at the time the captured data was recorded. More specifically, activity log data is structured data in which a caregiver records the daily care activities of an infant or toddler in chronological order, and may include data related to childcare, such as the child's physiological / behavioral characteristics (type of activity, state, and routine, etc.).

[0076] In addition, diary data may include diary data from the time the captured data was taken. More specifically, diary data may include data in which a guardian (or parent) emotionally or descriptively records the growth process and daily moments of an infant or toddler. Diary data may possess the characteristics of unstructured text data rather than a structured form, and may include the guardian's subjective interpretations and emotional responses.

[0077] In step (S120), the electronic picture frame system (100) can perform sentiment analysis based on the received diary data. The electronic picture frame system (100) can process the diary data using natural language processing. The electronic picture frame system (100) can preprocess the diary data by utilizing a natural language processing (NLP) model. For example, the electronic picture frame system (100) can perform morphological analysis, stop word removal, and sentence separation through a natural language processing model. The electronic picture frame system (100) can define the preprocessed diary data as natural language processing data.

[0078] The electronic photo frame system (100) can input preprocessed diary data into an emotion analysis model and output emotion analysis data. The emotion analysis model can classify the input diary data by emotion. For example, the emotion analysis model can output emotion analysis data including emotional states such as joy, anger, sadness, anxiety, hurt, embarrassment, worry, and interest. The emotion analysis model may include a deep learning-based KoBERT, KLUE-BERT, or a custom fine-tuned emotion classification model.

[0079] In step (S130), the electronic photo frame system (100) can extract keywords from shooting data and activity log data. For example, the electronic photo frame system (100) can extract and classify keywords by natural language processing of the activity log data and shooting data.

[0080] More specifically, the electronic photo frame system (100) can extract activity keywords by processing activity log data at the time the shooting data was captured using natural language processing. The electronic photo frame system (100) can select the most important keyword by comparing semantic similarity between a sentence and a keyword candidate, summarize a list of important keywords, or extract activity keywords based on pre-set core keywords.

[0081] Additionally, the electronic picture frame system (100) can extract image keywords by detecting objects through image analysis of captured data. The electronic picture frame system (100) can also filter only those with high reliability among the detected object types, or determine objects that appear frequently as important keywords.

[0082] In step (S140), the electronic photo frame system (100) can generate natural language data based on sentiment analysis data and keywords. In addition, the electronic photo frame system (100) can also generate natural language data by further including time stamp data and spatial data of the captured data.

[0083] For example, the electronic picture frame system (100) may include a natural language generation model such as a GPT family (e.g., KoGPT, GPT-2 fine-tuned, T5) or an LLM-based custom NLG model. The electronic picture frame system (100) may output natural language data in the form of narration based on the natural language generation model. The electronic picture frame system (100) may generate sentiment-based natural language data by structuring sentiment analysis data and keywords.

[0084] In step (S150), the electronic picture frame system (100) can generate voice data using the generated natural language data and transmit the generated voice data to the electronic picture frame for output.

[0085] The electronic picture frame system (100) can output voice data by using natural language data as input to a voice generation model. For example, the voice generation model can output voice data using a Text-to-Speech (TTS) method. The voice generation model can be formed by a combination of SV2TTS / Resemblyzer, Tacotron2, and WaveGlow.

[0086] The electronic picture frame system (100) can train a voice generation model to generate voice data from input natural language data as the speech of a specific person.

[0087] In addition, the output voice data can be set to be output in the spoken language of the voice model. That is, the voice data can be expressed in the first-person spoken language of a specific person learned. Thus, the user can be induced to have emotional connection through the voice data spoken via the electronic picture frame (20).

[0088] The actions in FIG. 4 are not limited in order, and additional actions may be performed between two adjacent actions. Additionally, at least some of the actions in FIG. 4 may be omitted.

[0089] FIG. 5 is a drawing illustrating an example of a user terminal and an electronic picture frame displayed according to an embodiment of the present invention.

[0090] Referring to FIG. 5, the user terminal (10) can store shooting data in the form of sharing an album. The user terminal (10) can easily replace the photos displayed on the electronic photo frame (20) according to themes, travel, users, age, etc.

[0091] Here, the user terminal (10) can input activity log data and diary data for each shooting data included in the album. The activity log data and diary data can be input in the form of text data.

[0092] The electronic photo frame (20) can display the captured data (I) provided from the user terminal (10) on the screen. Additionally, the electronic photo frame (20) can output voice data (V) through a speaker. That is, the electronic photo frame (20) can provide emotion-based voice data (V) corresponding to the captured data (I).

[0093] The electronic photo frame (20) can output together shooting data (I) and emotion-based voice data (V) formed based on activity log data and diary data included in the shooting data. By doing so, the electronic photo frame (20) can form a bond between family members by converting photos and videos into emotional content.

[0094] The features, structures, effects, etc. described in the embodiments above are included in at least one embodiment of the present invention and are not necessarily limited to only one embodiment. Furthermore, the features, structures, effects, etc. exemplified in each embodiment may be combined or modified and implemented in other embodiments by a person skilled in the art to which the embodiments belong. Accordingly, details regarding such combinations and modifications should be interpreted as being included within the scope of the present invention.

[0095] Furthermore, although the embodiments have been described above, this is merely illustrative and does not limit the invention. Those skilled in the art will understand that various modifications and applications not exemplified above are possible within the scope of the essential characteristics of the embodiments. In other words, each component specifically shown in the embodiments may be modified and implemented. Differences related to such modifications and applications should be interpreted as being included within the scope of the invention as defined in the appended claims. Explanation of the symbols

[0096] 100: Digital Photo Frame System 10: User terminal 20: Digital photo frame

Claims

Claim 1 A method for generating electronic photo frame content using emotion-based voice data comprises: receiving shooting data, activity log data, and diary data from a user terminal; analyzing emotions based on the diary data; extracting keywords based on the shooting data and the activity log data; generating natural language data based on the emotion analysis data and keywords; and generating voice data using the generated natural language data and transmitting the generated voice data to an electronic photo frame for output; wherein the emotion analysis step comprises: processing the diary data in natural language to generate natural language processing data; and inputting the natural language processing data into an emotion classification model to output emotion analysis data at a corresponding point in time; wherein the emotion classification model predicts the emotional state included in the emotion analysis data using a multi-class classification method; and the step of generating natural language data comprises inputting the emotion analysis data and the keywords into a natural language generation model to output natural language data, wherein the natural language data is output in a colloquial form from the perspective of the natural language generation model, and the input data of the natural language generation model may further include time stamp data at the time when the shooting data was taken and spatial data of the shooting data. Claim 2 delete Claim 3 A method for generating electronic photo frame content according to claim 1, wherein the keyword extraction step comprises: a step of extracting activity keywords based on activity log data at the time the shooting data was captured; a step of extracting image keywords by recognizing objects through image analysis of the shooting data; and a step of determining key keywords based on the extracted activity keywords and image keywords. Claim 4 delete Claim 5 delete Claim 6 A method for generating electronic photo frame content according to claim 1, wherein the step of generating and outputting voice data comprises: a step of generating voice data by using the natural language data as input to a voice generation model; a step of providing the generated voice data to a connected electronic photo frame; and a step of outputting the voice data received by the electronic photo frame and the captured data at the corresponding point in time of the voice data. Claim 7 In claim 6, the voice generation model is trained to collect user speech voice samples, extract features of the collected user speech voice samples into vectors, and synthesize them to output voice data, thereby creating an electronic picture frame content generation method. Claim 8 A computer-readable recording medium having a program stored on it for executing a method according to any one of paragraphs 1, 3, 6, and 7 on a computer. Claim 9 In an electronic photo frame system utilizing emotion-based voice data, the system comprises a communication unit; a processor; and a memory; wherein the processor comprises: a data receiving unit that receives shooting data, activity log data, and diary data from a user terminal; an emotion analysis unit that performs emotion analysis based on the diary data; a keyword extraction unit that extracts keywords based on the shooting data and the activity log data; a natural language data generation unit that generates natural language data based on the emotion analysis data and keywords; and a voice data generation unit that generates voice data using the generated natural language data; wherein the emotion analysis unit performs natural language processing on the diary data to generate natural language processing data, inputs the natural language processing data into an emotion classification model to output emotion analysis data at a corresponding point in time, and the emotion classification model predicts the emotion state included in the emotion analysis data using a multi-class classification method; wherein the natural language data generation unit inputs the emotion analysis data and the keywords into a natural language generation model to output natural language data, and the natural language data is output in a colloquial form from the perspective of the natural language generation model, and the input data of the natural language generation model further includes time stamp data at the time when the shooting data was captured and spatial data of the shooting data. Electronic picture frame system including Claim 10 delete Claim 11 In claim 9, the keyword extraction unit extracts activity keywords based on activity log data at the time the shooting data was captured, extracts image keywords by recognizing objects through image analysis of the shooting data, and determines key keywords based on the extracted activity keywords and image keywords in an electronic photo frame system. Claim 12 delete Claim 13 In claim 9, the voice data generation unit generates voice data by using the natural language data as input to a voice generation model and provides the generated voice data to a connected electronic photo frame, and the electronic photo frame outputs the voice data and the corresponding point-in-time shooting data of the voice data.

Images