System

Abstract

A system in accordance with an embodiment is directed to facilitating the generation, transformation, analysis, and utilization of media content to improve multilingual access and viewing experience.SOLUTION: A system according to an embodiment includes a speech-to-text transcriber, a re-vocalizer, a visual content generator, a clustering unit, and a synergy provider. The speech-to-text converter converts speech into text. The re-vocalization unit re-converts the text converted by the speech-to-text unit into speech. The visual content generation unit generates visual content. The clustering unit clusters and classifies the generated content. The synergy provision unit provides information related to a scene of a movie.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional technologies are complex when it comes to generating, converting, analyzing, and utilizing media content, leaving room for improvement in terms of multilingual access and improved viewing experiences.

[0005] The system according to the embodiment aims to facilitate the creation, conversion, analysis, and utilization of media content, and to improve multilingual access and viewing experiences. [Means for solving the problem]

[0006] The system according to the embodiment includes a speech-to-text unit, a re-speech unit, a visual content generation unit, a clustering unit, and a synergy providing unit. The speech-to-text unit converts speech into text. The re-speech unit converts the text converted by the speech-to-text unit back into speech. The visual content generation unit generates visual content. The clustering unit clusters and classifies the generated content. The synergy providing unit provides information related to movie scenes. [Effects of the Invention]

[0007] Systems according to embodiments can facilitate the creation, conversion, analysis, and utilization of media content, enhancing multilingual access and viewing experiences. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. DETAILED DESCRIPTION OF THE INVENTION

[0009] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0010] First, the terms used in the following description will be explained.

[0011] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit).

[0012] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0013] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0014] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), and Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0016] [First embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0020] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (for example, a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 (see FIG. 2) acquires the data indicating the user input.

[0021] Output device 40 includes a display 40A and a speaker 40B, and presents data to a user by outputting the data in a form of expression that the user can perceive (e.g., audio and / or text). Display 40A displays visible information such as text and images in accordance with instructions from processor 46. Speaker 40B outputs audio in accordance with instructions from processor 46. Camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0023] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0024] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0025] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0026] In the smart device 14, the specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the smart device 14 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0027] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (prediction result, etc.) using the data generation model 58 by communicating with the server device having the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device owned by a user (e.g., a mobile phone, a robot, a home appliance, etc.). Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example 1) Media Magic Concierge, an embodiment of the present invention, is an integrated platform that allows content creators, marketers, educators, and general users to easily create, convert, analyze, and utilize media content, enabling users to utilize media content in a variety of ways and obtain new viewing experiences.

[0029] The Media Magic Concierge according to the embodiment includes a speech-to-text unit, a re-voice unit, a visual content generation unit, a clustering unit, and a synergy providing unit. The speech-to-text unit converts speech into text. For example, it converts speech data into text data using a speech recognition algorithm. The speech-to-text unit can also convert speech into text using speech recognition software. The speech-to-text unit can also convert speech data into text in real time. The re-voice unit converts text back into speech. For example, it converts text data into speech data using speech synthesis technology. The re-voice unit can also convert text into speech using speech synthesis software. The re-voice unit can also convert text data into speech in real time. The visual content generation unit generates visual content. For example, it generates visual content using the DALL-E API. The visual content generation unit can also generate images based on user requests. The visual content generation unit can also generate visual content using generation AI. The clustering unit clusters and classifies the generated content. For example, it classifies images and text generated using an embedding function by theme. The clustering unit can also group highly relevant content using the generation AI. The clustering unit can also classify content using a clustering algorithm. The synergy providing unit provides information related to movie scenes. For example, when a viewer requests images or information related to a certain scene in a movie, the generation AI generates highly relevant images and provides them with commentary. The synergy providing unit can also provide historical background related to the movie scene. The synergy providing unit can also provide related information based on the viewer's request. This allows the Media Magic Concierge according to the embodiment to enable users to utilize media content in a variety of ways and achieve new viewing experiences. For example, educators can provide teaching materials in multiple languages, and marketers can create visually appealing advertisements.In addition, general users can easily obtain information related to movie scenes.

[0030] The speech-to-text unit can automatically remove background noise from the speech and generate clear speech data. The speech-to-text unit, for example, develops an algorithm that automatically removes background noise from the speech data and generates clear speech data. This improves the accuracy of the text conversion. The speech-to-text unit also uses noise removal technology for the speech data to reduce background noise and generate clear speech data. This improves the accuracy of the text conversion. The speech-to-text unit also applies a noise removal algorithm to the speech data to automatically remove background noise. This improves the accuracy of the text conversion. This improves the accuracy of the text conversion.

[0031] The speech-to-text unit can automatically identify different dialects and accents and convert them into standard Japanese. The speech-to-text unit, for example, analyzes speech data and develops an algorithm that automatically identifies different dialects and accents, thereby converting them into standard Japanese. The speech-to-text unit also builds a system that automatically identifies dialects and accents in speech data and converts them into standard Japanese, thereby improving the accuracy of the text conversion. The speech-to-text unit also introduces technology that automatically identifies dialects and accents in speech data and converts them into standard Japanese, thereby improving the accuracy of the text conversion. This makes it possible to convert different dialects and accents into standard Japanese.

[0032] The speech-to-text unit can automatically recognize technical terms and industry jargon and provide appropriate translations. The speech-to-text unit, for example, analyzes speech data and develops an algorithm that automatically recognizes technical terms and industry jargon. This provides appropriate translations. The speech-to-text unit also builds a system that automatically recognizes technical terms and industry jargon in speech data and provides appropriate translations. This improves the accuracy of the text conversion. The speech-to-text unit also introduces technology that automatically recognizes technical terms and industry jargon in speech data and provides appropriate translations. This improves the accuracy of the text conversion. This enables appropriate translations of technical terms and industry jargon.

[0033] The visual content generation unit can automatically generate detailed metadata for the generated visual content, thereby improving searchability. The visual content generation unit, for example, develops an algorithm for automatically generating detailed metadata for the generated visual content, thereby improving searchability. The visual content generation unit also builds a system for automatically generating detailed metadata for the generated visual content, thereby improving searchability. The visual content generation unit also introduces technology for automatically generating detailed metadata for the generated visual content, thereby improving searchability. This improves searchability for the visual content.

[0034] The visual content generation unit can automatically generate 3D models and animations when generating visual content. For example, the visual content generation unit adds a function for automatically generating 3D models when generating visual content. For example, it generates a 3D model of an object requested by a user. The visual content generation unit also adds a function for automatically generating animations when generating visual content. For example, it generates an animation of a scene requested by a user. The visual content generation unit also introduces technology for automatically generating 3D models and animations when generating visual content. This improves visual expressiveness. This allows the automatic generation of 3D models and animations when generating visual content.

[0035] The visual content generation unit can generate diverse images that incorporate elements of different cultures and regions. For example, the visual content generation unit generates images that incorporate elements of different cultures and regions when generating visual content. For example, it generates images that reflect the cultural background requested by the user. Furthermore, the visual content generation unit generates images that incorporate elements of different regions when generating visual content. For example, it generates images that reflect the scenery of the region requested by the user. Furthermore, the visual content generation unit introduces technology for generating diverse images that incorporate elements of different cultures and regions when generating visual content. This improves visual diversity. This makes it possible to generate diverse images that incorporate elements of different cultures and regions.

[0036] The clustering unit can improve the clustering algorithm and take into account the temporal relevance between pieces of content in order to measure the relevance of content with higher accuracy. The clustering unit, for example, improves the clustering algorithm and takes into account the temporal relevance between pieces of content. For example, pieces of content created around the same time are grouped into the same cluster. The clustering unit also improves the clustering algorithm and takes into account the temporal relevance between pieces of content in order to measure the relevance of content with higher accuracy. For example, it performs clustering along a time axis. The clustering unit also improves the clustering algorithm and takes into account the temporal relevance between pieces of content. For example, it measures the relevance between past content and current content. This makes it possible to measure the relevance of content with higher accuracy.

[0037] The clustering unit can generate an interactive cluster map to visualize the clustering results and enable the user to intuitively understand them. For example, the clustering unit visualizes the clustering results as an interactive cluster map, allowing the user to access detailed information by clicking and zooming. For example, each cluster may be displayed as a node, and related ideas may be connected by links. The clustering unit also allows the user to intuitively understand the contents and relationships of each cluster by manipulating the cluster map. For example, a function may be added to display a summary of an idea by hovering the mouse over it. The clustering unit also updates the cluster map in real time to instantly reflect newly added or changed ideas. For example, a function may be provided to allow the user to re-cluster related ideas by moving a cluster. This allows the clustering results to be intuitively understood.

[0038] The clustering unit can automatically detect relationships between different clusters and generate hyperlinks that connect related clusters. The clustering unit, for example, analyzes the clustering results and develops an algorithm that automatically detects relationships between different clusters. For example, it measures the relationships based on commonalities in keywords or topics. The clustering unit also generates hyperlinks that connect highly related clusters, allowing users to easily access related ideas. For example, it allows users to move between related clusters with just one click. The clustering unit also uses hyperlinks to build a system that integrates information between different clusters and gains new insights. For example, it displays related ideas in a single view to grasp the overall picture. This allows the system to automatically detect relationships between different clusters and generate hyperlinks.

[0039] The clustering unit can integrate clustering results with other datasets to gain new insights. For example, the clustering unit can integrate clustered ideas with patent data to analyze technological trends and competitive situations. For example, it can correlate and display patent information in the same technical field. The clustering unit can also combine market data with clustering results to discover new business opportunities. For example, it can extract ideas from clusters that address specific market needs. The clustering unit can also integrate clustering results with other datasets to understand the latest research trends and technological innovations. For example, it can link related papers and research results to clusters. This allows the clustering results to be integrated with other datasets to gain new insights.

[0040] The synergy providing unit can refer to the viewer's past viewing history and provide personalized information when providing information related to a movie scene. The synergy providing unit, for example, refers to the viewer's past viewing history and provides personalized information when providing information related to a movie scene. For example, it provides information related to movies that the viewer has previously watched. Furthermore, the synergy providing unit builds a system that refers to the viewer's past viewing history and provides personalized information when providing information related to a movie scene. Furthermore, the synergy providing unit introduces technology that refers to the viewer's past viewing history and provides personalized information when providing information related to a movie scene. This makes it possible to provide personalized information based on the viewer's past viewing history.

[0041] The synergy providing unit can monitor the viewer's real-time reactions when providing information related to a movie scene and provide the information at an appropriate timing. For example, the synergy providing unit can monitor the viewer's real-time reactions when providing information related to a movie scene and provide the information at an appropriate timing. For example, the synergy providing unit can provide related information when the viewer is excited. Furthermore, the synergy providing unit builds a system that monitors the viewer's real-time reactions when providing information related to a movie scene and provides the information at an appropriate timing. Furthermore, the synergy providing unit introduces technology that monitors the viewer's real-time reactions when providing information related to a movie scene and provides the information at an appropriate timing. This makes it possible to provide information at an appropriate timing based on the viewer's real-time reactions.

[0042] The synergy providing unit can provide not only visual content but also audio and text information at the same time when providing information related to a movie scene. For example, when providing information related to a movie scene, the synergy providing unit can provide not only visual content but also audio and text information at the same time. For example, a scene commentary can be provided in audio. The synergy providing unit also builds a system that provides not only visual content but also audio and text information at the same time when providing information related to a movie scene. The synergy providing unit also introduces technology that provides not only visual content but also audio and text information at the same time when providing information related to a movie scene. This makes it possible to provide not only visual content but also audio and text information at the same time.

[0043] The synergy providing unit can provide highly relevant information by cross-referencing movies of different genres and themes when providing information related to a movie scene. The synergy providing unit, for example, can provide highly relevant information by cross-referencing movies of different genres and themes when providing information related to a movie scene. For example, the synergy providing unit can provide information about other movies with the same theme. The synergy providing unit also builds a system that provides highly relevant information by cross-referencing movies of different genres and themes when providing information related to a movie scene. The synergy providing unit also introduces technology that provides highly relevant information by cross-referencing movies of different genres and themes when providing information related to a movie scene. This makes it possible to provide highly relevant information by cross-referencing movies of different genres and themes.

[0044] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0045] Media Magic Concierge can also analyze a user's behavioral history to make personalized content suggestions. For example, it can suggest new content that a user might be interested in based on the genre and theme of content they have previously viewed. It can also provide content at the optimal time by taking into account the user's viewing time and frequency. It can also collect user feedback and continuously improve its suggestions. This allows users to efficiently find content that suits them.

[0046] MediaMagic Concierge also enables seamless content sharing across devices. For example, you can continue watching content on your smartphone on your TV or tablet. It also synchronizes viewing history and settings even when users use multiple devices. It also provides an interface for easily transferring content between devices. This allows users to enjoy a consistent viewing experience on any device.

[0047] Media Magic Concierge can also automatically generate customized playlists based on user preferences. For example, if a user likes a particular genre or artist, it can create a playlist based on that information. It can also analyze a user's viewing history and ratings to select the most suitable content. It can also incorporate user feedback and continuously improve the playlist content. This allows users to efficiently enjoy content that suits their preferences.

[0048] Media Magic Concierge can also provide a platform for users to share content they generate with other users. For example, users can easily upload videos and images they have created and share them with other users. It can also provide a function for users to comment on and rate the shared content. It can also add a function for promoting new collaborations based on the shared content. This allows users to widely share their content and enjoy interacting with other users.

[0049] Media Magic Concierge can also provide a function to evaluate content quality based on users' viewing history and ratings. For example, it can prioritize the display of content that users have given high ratings. It can also filter out low-rated content. It can also collect user feedback and continuously improve content quality ratings. This allows users to find high-quality content efficiently.

[0050] The processing flow of the first embodiment will be briefly explained below.

[0051] Step 1: The speech-to-text unit converts speech to text. For example, the speech-to-text unit converts speech data to text data using a speech recognition algorithm. The speech-to-text unit can also convert speech to text using speech recognition software. The speech-to-text unit can also convert speech data to text in real time. Step 2: The re-speech unit converts the text back into speech. For example, the re-speech unit converts the text data into speech data using speech synthesis technology. The re-speech unit can also convert the text into speech using speech synthesis software. The re-speech unit can also convert the text data into speech in real time. Step 3: The visual content generator generates visual content. For example, the visual content may be generated using the DALL-E API. The visual content generator may also generate images based on user requests. The visual content generator may also generate visual content using generative AI. Step 4: The clustering unit clusters and classifies the generated content. For example, it uses the embedding function to classify images and text generated by theme. The clustering unit can also use generation AI to group highly related content. The clustering unit can also classify content using a clustering algorithm. Step 5: The synergy provider provides information related to a scene in the movie. For example, if a viewer requests images or information related to a scene in the movie, the generation AI generates a highly relevant image and provides it with an explanation. The synergy provider can also provide historical background related to the scene in the movie. The synergy provider can also provide related information based on the viewer's request.

[0052] (Example 2) Media Magic Concierge, an embodiment of the present invention, is an integrated platform that allows content creators, marketers, educators, and general users to easily create, convert, analyze, and utilize media content, enabling users to utilize media content in a variety of ways and obtain new viewing experiences.

[0053] The Media Magic Concierge according to the embodiment includes a speech-to-text unit, a re-voice unit, a visual content generation unit, a clustering unit, and a synergy providing unit. The speech-to-text unit converts speech into text. For example, it converts speech data into text data using a speech recognition algorithm. The speech-to-text unit can also convert speech into text using speech recognition software. The speech-to-text unit can also convert speech data into text in real time. The re-voice unit converts text back into speech. For example, it converts text data into speech data using speech synthesis technology. The re-voice unit can also convert text into speech using speech synthesis software. The re-voice unit can also convert text data into speech in real time. The visual content generation unit generates visual content. For example, it generates visual content using the DALL-E API. The visual content generation unit can also generate images based on user requests. The visual content generation unit can also generate visual content using generation AI. The clustering unit clusters and classifies the generated content. For example, it classifies images and text generated using an embedding function by theme. The clustering unit can also group highly relevant content using the generation AI. The clustering unit can also classify content using a clustering algorithm. The synergy providing unit provides information related to movie scenes. For example, when a viewer requests images or information related to a certain scene in a movie, the generation AI generates highly relevant images and provides them with commentary. The synergy providing unit can also provide historical background related to the movie scene. The synergy providing unit can also provide related information based on the viewer's request. This allows the Media Magic Concierge according to the embodiment to enable users to utilize media content in a variety of ways and achieve new viewing experiences. For example, educators can provide teaching materials in multiple languages, and marketers can create visually appealing advertisements.In addition, general users can easily obtain information related to movie scenes.

[0054] The speech-to-text conversion unit can perform emotional analysis of the speech data and perform text conversion according to the emotion. The speech-to-text conversion unit, for example, analyzes the speech data and performs emotional analysis. For example, speech containing an emotion of anger is converted into emphasized text. The expression of the text is adjusted based on the emotion score. The speech-to-text conversion unit also performs emotional analysis of the speech data and performs text conversion according to the emotion. For example, speech containing an emotion of sadness is converted into text with a softer expression. The speech-to-text conversion unit also performs emotional analysis of the speech data and performs text conversion according to the emotion. For example, speech containing an emotion of joy is converted into text with a brighter expression. This makes it possible to perform text conversion according to the emotion.

[0055] The speech-to-text unit can automatically remove background noise from the speech and generate clear speech data. The speech-to-text unit, for example, develops an algorithm that automatically removes background noise from the speech data and generates clear speech data. This improves the accuracy of the text conversion. The speech-to-text unit also uses noise removal technology for the speech data to reduce background noise and generate clear speech data. This improves the accuracy of the text conversion. The speech-to-text unit also applies a noise removal algorithm to the speech data to automatically remove background noise. This improves the accuracy of the text conversion. This improves the accuracy of the text conversion.

[0056] The re-speech unit can use the emotion estimation function to adjust the voice tone according to the user's emotion during re-speech. The re-speech unit, for example, uses the emotion estimation function to adjust the voice tone according to the user's emotion during re-speech. For example, if the emotion of joy is included, the voice tone is re-spoken in a brighter tone. The re-speech unit also uses the emotion estimation function to adjust the voice tone according to the user's emotion during re-speech. For example, if the emotion of anger is included, the voice tone is re-spoken in an emphasized tone. The re-speech unit also uses the emotion estimation function to adjust the voice tone according to the user's emotion during re-speech. For example, if the emotion of sadness is included, the voice tone is re-spoken in a softer tone. This makes it possible to adjust the voice tone according to the user's emotion during re-speech.

[0057] The speech-to-text unit can automatically identify different dialects and accents and convert them into standard Japanese. The speech-to-text unit, for example, analyzes speech data and develops an algorithm that automatically identifies different dialects and accents, thereby converting them into standard Japanese. The speech-to-text unit also builds a system that automatically identifies dialects and accents in speech data and converts them into standard Japanese, thereby improving the accuracy of the text conversion. The speech-to-text unit also introduces technology that automatically identifies dialects and accents in speech data and converts them into standard Japanese, thereby improving the accuracy of the text conversion. This makes it possible to convert different dialects and accents into standard Japanese.

[0058] The speech-to-text unit can automatically recognize technical terms and industry jargon and provide appropriate translations. The speech-to-text unit, for example, analyzes speech data and develops an algorithm that automatically recognizes technical terms and industry jargon. This provides appropriate translations. The speech-to-text unit also builds a system that automatically recognizes technical terms and industry jargon in speech data and provides appropriate translations. This improves the accuracy of the text conversion. The speech-to-text unit also introduces technology that automatically recognizes technical terms and industry jargon in speech data and provides appropriate translations. This improves the accuracy of the text conversion. This enables appropriate translations of technical terms and industry jargon.

[0059] The speech-to-text unit can analyze the emotions of the user when inputting speech in real time and provide appropriate feedback. The speech-to-text unit, for example, uses an emotion estimation function to analyze the emotions of the user when inputting speech in real time. This provides appropriate feedback. Furthermore, the speech-to-text unit uses the emotion estimation function to analyze the emotions of the user when inputting speech in real time, and builds a system that provides appropriate feedback. Furthermore, the speech-to-text unit introduces technology that uses the emotion estimation function to analyze the emotions of the user when inputting speech in real time and provides appropriate feedback. This makes it possible to provide feedback in real time that corresponds to the user's emotions.

[0060] The visual content generation unit can perform emotion analysis on the generated visual content and filter it according to the emotion. For example, the visual content generation unit performs emotion analysis on the generated visual content and preferentially displays images that evoke positive emotions. The visual content generation unit also performs emotion analysis on the generated visual content and filters out images that evoke negative emotions. The visual content generation unit also performs emotion analysis on the generated visual content and filters it according to the emotion. For example, if you want to relax, preferentially display images of natural landscapes. This makes it possible to filter the visual content according to the emotion.

[0061] The visual content generation unit can automatically generate detailed metadata for the generated visual content, thereby improving searchability. The visual content generation unit, for example, develops an algorithm for automatically generating detailed metadata for the generated visual content, thereby improving searchability. The visual content generation unit also builds a system for automatically generating detailed metadata for the generated visual content, thereby improving searchability. The visual content generation unit also introduces technology for automatically generating detailed metadata for the generated visual content, thereby improving searchability. This improves searchability for the visual content.

[0062] The visual content generation unit can use the emotion estimation function to generate visual content according to the user's emotion. The visual content generation unit, for example, uses the emotion estimation function to generate visual content according to the user's emotion. For example, if the user wants to relax, it generates an image of a natural landscape. The visual content generation unit also uses the emotion estimation function to generate visual content according to the user's emotion. For example, if the user feels a strong emotion of joy, it generates an image with bright colors. The visual content generation unit also uses the emotion estimation function to generate visual content according to the user's emotion. For example, if the user feels a strong emotion of sadness, it generates an image with soft colors. In this way, visual content according to the user's emotion can be generated.

[0063] The visual content generation unit can automatically generate 3D models and animations when generating visual content. For example, the visual content generation unit adds a function for automatically generating 3D models when generating visual content. For example, it generates a 3D model of an object requested by a user. The visual content generation unit also adds a function for automatically generating animations when generating visual content. For example, it generates an animation of a scene requested by a user. The visual content generation unit also introduces technology for automatically generating 3D models and animations when generating visual content. This improves visual expressiveness. This allows the automatic generation of 3D models and animations when generating visual content.

[0064] The visual content generation unit can generate diverse images that incorporate elements of different cultures and regions. For example, the visual content generation unit generates images that incorporate elements of different cultures and regions when generating visual content. For example, it generates images that reflect the cultural background requested by the user. Furthermore, the visual content generation unit generates images that incorporate elements of different regions when generating visual content. For example, it generates images that reflect the scenery of the region requested by the user. Furthermore, the visual content generation unit introduces technology for generating diverse images that incorporate elements of different cultures and regions when generating visual content. This improves visual diversity. This makes it possible to generate diverse images that incorporate elements of different cultures and regions.

[0065] The visual content generation unit uses the emotion estimation function to analyze the emotion of the user when requesting visual content and can suggest optimal content. For example, the visual content generation unit uses the emotion estimation function to analyze the emotion of the user when requesting visual content. This allows the optimal content to be suggested. Furthermore, the visual content generation unit uses the emotion estimation function to build a system that analyzes the emotion of the user when requesting visual content and suggests optimal content. Furthermore, the visual content generation unit introduces technology that uses the emotion estimation function to analyze the emotion of the user when requesting visual content and suggests optimal content. This allows the optimal visual content to be suggested according to the user's emotion.

[0066] The clustering unit can perform sentiment analysis on the clustered content and perform clustering according to the sentiment. For example, the clustering unit performs sentiment analysis on the clustered content and groups content with positive sentiment into one cluster. The clustering unit also performs sentiment analysis on the clustered content and groups content with negative sentiment into another cluster. The clustering unit also performs sentiment analysis on the clustered content and performs clustering according to the sentiment. For example, content with a joyful sentiment is grouped into one cluster. This makes it possible to perform clustering according to the sentiment.

[0067] The clustering unit can improve the clustering algorithm and take into account the temporal relevance between pieces of content in order to measure the relevance of content with higher accuracy. The clustering unit, for example, improves the clustering algorithm and takes into account the temporal relevance between pieces of content. For example, pieces of content created around the same time are grouped into the same cluster. The clustering unit also improves the clustering algorithm and takes into account the temporal relevance between pieces of content in order to measure the relevance of content with higher accuracy. For example, it performs clustering along a time axis. The clustering unit also improves the clustering algorithm and takes into account the temporal relevance between pieces of content. For example, it measures the relevance between past content and current content. This makes it possible to measure the relevance of content with higher accuracy.

[0068] The clustering unit can generate an interactive cluster map to visualize the clustering results and enable the user to intuitively understand them. For example, the clustering unit visualizes the clustering results as an interactive cluster map, allowing the user to access detailed information by clicking and zooming. For example, each cluster may be displayed as a node, and related ideas may be connected by links. The clustering unit also allows the user to intuitively understand the contents and relationships of each cluster by manipulating the cluster map. For example, a function may be added to display a summary of an idea by hovering the mouse over it. The clustering unit also updates the cluster map in real time to instantly reflect newly added or changed ideas. For example, a function may be provided to allow the user to re-cluster related ideas by moving a cluster. This allows the clustering results to be intuitively understood.

[0069] The clustering unit can automatically detect relationships between different clusters and generate hyperlinks that connect related clusters. The clustering unit, for example, analyzes the clustering results and develops an algorithm that automatically detects relationships between different clusters. For example, it measures the relationships based on commonalities in keywords or topics. The clustering unit also generates hyperlinks that connect highly related clusters, allowing users to easily access related ideas. For example, it allows users to move between related clusters with just one click. The clustering unit also uses hyperlinks to build a system that integrates information between different clusters and gains new insights. For example, it displays related ideas in a single view to grasp the overall picture. This allows the system to automatically detect relationships between different clusters and generate hyperlinks.

[0070] The clustering unit can integrate clustering results with other datasets to gain new insights. For example, the clustering unit can integrate clustered ideas with patent data to analyze technological trends and competitive situations. For example, it can correlate and display patent information in the same technical field. The clustering unit can also combine market data with clustering results to discover new business opportunities. For example, it can extract ideas from clusters that address specific market needs. The clustering unit can also integrate clustering results with other datasets to understand the latest research trends and technological innovations. For example, it can link related papers and research results to clusters. This allows the clustering results to be integrated with other datasets to gain new insights.

[0071] The clustering unit uses the emotion estimation function to analyze users' emotional reactions to the clustered ideas and can identify clusters that are likely to be emotionally relatable. The clustering unit, for example, collects users' emotional reactions to the clustered ideas in real time and identifies clusters that are likely to be emotionally relatable based on the data. For example, clusters with a high number of positive emotional reactions are preferentially displayed. The clustering unit also uses the emotion estimation function to analyze users' emotional scores for clusters and identify clusters with high emotional value. For example, clusters with high emotional scores are prominently displayed. The clustering unit also identifies clusters that are likely to be emotionally relatable based on the user's emotional reaction data and generates new ideas centered around those clusters. For example, ideas related to clusters with high relatability are automatically generated. This makes it possible to identify clusters that are likely to be emotionally relatable.

[0072] When providing information related to a movie scene, the synergy providing unit can use the emotion estimation function to provide information that corresponds to the viewer's emotion. For example, when providing information related to a movie scene, the synergy providing unit uses the emotion estimation function to provide information that corresponds to the viewer's emotion. For example, for a moving scene, it provides information that deepens the emotion. Furthermore, when providing information related to a movie scene, the synergy providing unit uses the emotion estimation function to provide information that corresponds to the viewer's emotion. For example, for a tense scene, it provides information that increases the tension. Furthermore, when providing information related to a movie scene, the synergy providing unit uses the emotion estimation function to provide information that corresponds to the viewer's emotion. For example, for a happy scene, it provides information that amplifies the joy. In this way, information that corresponds to the viewer's emotion can be provided.

[0073] The synergy providing unit can refer to the viewer's past viewing history and provide personalized information when providing information related to a movie scene. The synergy providing unit, for example, refers to the viewer's past viewing history and provides personalized information when providing information related to a movie scene. For example, it provides information related to movies that the viewer has previously watched. Furthermore, the synergy providing unit builds a system that refers to the viewer's past viewing history and provides personalized information when providing information related to a movie scene. Furthermore, the synergy providing unit introduces technology that refers to the viewer's past viewing history and provides personalized information when providing information related to a movie scene. This makes it possible to provide personalized information based on the viewer's past viewing history.

[0074] The synergy providing unit can monitor the viewer's real-time reactions when providing information related to a movie scene and provide the information at an appropriate timing. For example, the synergy providing unit can monitor the viewer's real-time reactions when providing information related to a movie scene and provide the information at an appropriate timing. For example, the synergy providing unit can provide related information when the viewer is excited. Furthermore, the synergy providing unit builds a system that monitors the viewer's real-time reactions when providing information related to a movie scene and provides the information at an appropriate timing. Furthermore, the synergy providing unit introduces technology that monitors the viewer's real-time reactions when providing information related to a movie scene and provides the information at an appropriate timing. This makes it possible to provide information at an appropriate timing based on the viewer's real-time reactions.

[0075] The synergy providing unit can provide not only visual content but also audio and text information at the same time when providing information related to a movie scene. For example, when providing information related to a movie scene, the synergy providing unit can provide not only visual content but also audio and text information at the same time. For example, a scene commentary can be provided in audio. The synergy providing unit also builds a system that provides not only visual content but also audio and text information at the same time when providing information related to a movie scene. The synergy providing unit also introduces technology that provides not only visual content but also audio and text information at the same time when providing information related to a movie scene. This makes it possible to provide not only visual content but also audio and text information at the same time.

[0076] The synergy providing unit can provide highly relevant information by cross-referencing movies of different genres and themes when providing information related to a movie scene. The synergy providing unit, for example, can provide highly relevant information by cross-referencing movies of different genres and themes when providing information related to a movie scene. For example, the synergy providing unit can provide information about other movies with the same theme. The synergy providing unit also builds a system that provides highly relevant information by cross-referencing movies of different genres and themes when providing information related to a movie scene. The synergy providing unit also introduces technology that provides highly relevant information by cross-referencing movies of different genres and themes when providing information related to a movie scene. This makes it possible to provide highly relevant information by cross-referencing movies of different genres and themes.

[0077] The synergy providing unit can use the emotion estimation function to analyze the emotion a viewer feels toward a movie scene and provide information based on that emotion. The synergy providing unit, for example, uses the emotion estimation function to analyze the emotion a viewer feels toward a movie scene and provide information based on that emotion. For example, for a moving scene, information that deepens the emotion is provided. The synergy providing unit also uses the emotion estimation function to build a system that analyzes the emotion a viewer feels toward a movie scene and provides information based on that emotion. The synergy providing unit also introduces technology that uses the emotion estimation function to analyze the emotion a viewer feels toward a movie scene and provides information based on that emotion. This makes it possible to provide information based on the viewer's emotion.

[0078] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0079] Media Magic Concierge can also analyze a user's behavioral history to make personalized content suggestions. For example, it can suggest new content that a user might be interested in based on the genre and theme of content they have previously viewed. It can also provide content at the optimal time by taking into account the user's viewing time and frequency. It can also collect user feedback and continuously improve its suggestions. This allows users to efficiently find content that suits them.

[0080] Media Magic Concierge can also estimate the user's emotions and suggest appropriate content based on the estimated emotions. For example, if the user is feeling stressed, it can suggest relaxing content. If the user is feeling happy, it can suggest content that will further enhance that emotion. It can also monitor changes in the user's emotions in real time and switch content at the appropriate time. This allows users to enjoy content that matches their emotions.

[0081] MediaMagic Concierge also enables seamless content sharing across devices. For example, you can continue watching content on your smartphone on your TV or tablet. It also synchronizes viewing history and settings even when users use multiple devices. It also provides an interface for easily transferring content between devices. This allows users to enjoy a consistent viewing experience on any device.

[0082] Media Magic Concierge can also estimate the user's emotions and adjust the color tone and design of visual content based on the estimated emotions. For example, if the user wants to relax, it can provide visual content with soft colors. On the other hand, if the user is feeling energetic, it can provide visual content with bright colors. Furthermore, it can adjust the design of visual content in real time according to changes in the user's emotions. This allows users to enjoy visual content that matches their emotions.

[0083] Media Magic Concierge can also automatically generate customized playlists based on user preferences. For example, if a user likes a particular genre or artist, it can create a playlist based on that information. It can also analyze a user's viewing history and ratings to select the most suitable content. It can also incorporate user feedback and continuously improve the playlist content. This allows users to efficiently enjoy content that suits their preferences.

[0084] Media Magic Concierge can also estimate the user's emotions and adjust the tone and tempo of the audio content based on the estimated emotions. For example, if the user wants to relax, it can provide audio content with a gentle tone and slow tempo. On the other hand, if the user wants to concentrate, it can provide audio content with a clear tone and fast tempo. Furthermore, it can adjust the tone and tempo of the audio content in real time according to changes in the user's emotions. This allows users to enjoy audio content that matches their emotions.

[0085] Media Magic Concierge can also provide a platform for users to share content they generate with other users. For example, users can easily upload videos and images they have created and share them with other users. It can also provide a function for users to comment on and rate the shared content. It can also add a function for promoting new collaborations based on the shared content. This allows users to widely share their content and enjoy interacting with other users.

[0086] Media Magic Concierge can also estimate the user's emotions and adjust the content display order based on the estimated emotions. For example, if the user is feeling stressed, it can prioritize relaxing content. On the other hand, if the user is feeling energetic, it can prioritize active content. Furthermore, it can adjust the display order in real time according to changes in the user's emotions. This allows users to efficiently find content that matches their emotions.

[0087] Media Magic Concierge can also provide a function to evaluate content quality based on users' viewing history and ratings. For example, it can prioritize the display of content that users have given high ratings. It can also filter out low-rated content. It can also collect user feedback and continuously improve content quality ratings. This allows users to find high-quality content efficiently.

[0088] Media Magic Concierge can also estimate a user's emotions and provide a function to explain the reasons for recommending content based on the estimated emotions. For example, if a user wants to relax, it can display an explanation such as "This content has a relaxing effect." Or, if a user is feeling energetic, it can display an explanation such as "This content will increase your energy." Furthermore, it can adjust the reasons for recommendations in real time according to changes in the user's emotions. This makes it easier for users to understand content that matches their emotions.

[0089] The processing flow of the second embodiment will be briefly explained below.

[0090] Step 1: The speech-to-text unit converts speech to text. For example, the speech-to-text unit converts speech data to text data using a speech recognition algorithm. The speech-to-text unit can also convert speech to text using speech recognition software. The speech-to-text unit can also convert speech data to text in real time. Step 2: The re-speech unit converts the text back into speech. For example, the re-speech unit converts the text data into speech data using speech synthesis technology. The re-speech unit can also convert the text into speech using speech synthesis software. The re-speech unit can also convert the text data into speech in real time. Step 3: The visual content generator generates visual content. For example, the visual content may be generated using the DALL-E API. The visual content generator may also generate images based on user requests. The visual content generator may also generate visual content using generative AI. Step 4: The clustering unit clusters and classifies the generated content. For example, it uses the embedding function to classify images and text generated by theme. The clustering unit can also use generation AI to group highly related content. The clustering unit can also classify content using a clustering algorithm. Step 5: The synergy provider provides information related to a scene in the movie. For example, if a viewer requests images or information related to a scene in the movie, the generation AI generates a highly relevant image and provides it with an explanation. The synergy provider can also provide historical background related to the scene in the movie. The synergy provider can also provide related information based on the viewer's request.

[0091] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0092] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AIs include the data generation model 58, such as a neural network model (e.g., a neural network model), and a neural network model (e.g., a neural network model). The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating speech, text data indicating text, and image data indicating an image is also input to the data generation model 58. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0093] Furthermore, the processing by the data processing system 10 described above is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0094] [Second embodiment] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0095] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0096] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0097] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0098] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0099] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0100] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0101] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0102] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0103] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0104] In the smart glasses 214, the specific processing is performed by the processor 46. A specific processing program 60 is stored in the storage 50. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. The smart glasses 214 also have a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0105] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0106] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0107] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0108] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or an external device, etc., and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0109] [Third embodiment] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0110] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0111] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0112] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0113] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0114] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0115] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0116] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0117] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0118] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0119] In the headset type terminal 314, the specific processing is performed by the processor 46. A specific processing program 60 is stored in the storage 50. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the headset type terminal 314 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0120] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0121] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0122] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0123] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset type terminal 314, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset type terminal 314. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the headset type terminal 314 or an external device, etc., and the headset type terminal 314 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0124] [Fourth embodiment] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0125] 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0126] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0127] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0128] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0129] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS image sensor or a CCD image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0130] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0131] The control object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0132] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0133] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0134] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0135] In the robot 414, the specific processing is performed by the processor 46. A specific processing program 60 is stored in the storage 50. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. The robot 414 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0136] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0137] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0138] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0139] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or an external device, etc., and the robot 414 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0140] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0141] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion encompasses both emotions and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0142] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0143] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[0144] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is expressed, and when they approach the ideal, a state of pleasure is expressed. Emotions can also be created for robots, cars, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is expressed, and when they approach the ideal, a state of pleasure is expressed. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on speech emotion recognition and brain physiological signal analysis systems for emotions, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the area called "reaction," where sensation is dominant. The right half of the emotion map lists emotions belonging to the area called "situation," where situational awareness is dominant.

[0145] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[0146] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[0147] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers including computer 22.

[0148] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0149] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0150] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0151] The hardware resource for executing a specific process can be any of the following types of processors: A processor, for example, is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. A processor also includes a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[0152] The hardware resource that executes the specific process may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[0153] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0154] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[0155] In the above example, the first to fourth embodiments have been described separately, but some or all of these embodiments may be combined. The smart device 14, smart glasses 214, headset terminal 314, and robot 414 are merely examples, and they may be combined, or other devices may be used. In the above example, the first and second embodiments have been described separately, but they may be combined.

[0156] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0157] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0158] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot

Claims

1. a speech-to-text unit that converts speech into text; a re-speech conversion unit that converts the text converted by the speech-to-text conversion unit back into speech; a visual content generator for generating visual content; a clustering unit for clustering and classifying the generated content; a synergy providing unit that provides information related to scenes in the movie. A system characterized by:

2. The speech-to-text unit Performs sentiment analysis on voice data and converts it into text according to the sentiment 2. The system of claim 1.

3. The speech-to-text unit Automatically remove background noise from the audio to generate clear audio data 2. The system of claim 1.

4. The re-voice conversion unit Adjusting the voice tone according to the user's emotion during re-speech 2. The system of claim 1.

5. The speech-to-text unit Automatically identifies different dialects and accents and converts them into standard Japanese 2. The system of claim 1.

6. The speech-to-text unit Automatically recognize technical and industry jargon and provide appropriate translations 2. The system of claim 1.

7. The speech-to-text unit Analyzes emotions in real time as users input voice and provides appropriate feedback 2. The system of claim 1.

8. The visual content generator Sentiment analysis is performed on the generated visual content, and filtering is performed according to the emotion.

2. The system of claim 1.

Images