System

Abstract

To provide a system which analyzes an image and voice data, and which automatically detects a statement that has a risk of backlash and deals with it properly.SOLUTION: A system includes a reception part, a generation part, a detection part, and an edition part. The reception part inputs an image or voice data. The generation part analyses the data inputted by the reception part, and generates a subtitle. The detection part detects a statement that has a risk of backlash based on the subtitle generated by the generation part. The edition part cuts a statement or inserts a telop based on the statement detected by the detection part.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional technology does not adequately detect comments that pose a risk of causing an uproar automatically and deal with them appropriately when analyzing and editing video and audio data, so there is room for improvement.

[0005] The system according to the embodiment aims to analyze video and audio data, automatically detect comments that pose a risk of sparking an online backlash, and take appropriate action. [Means for solving the problem]

[0006] The system according to the embodiment includes a reception unit, a generation unit, a detection unit, and an editing unit. The reception unit inputs video or audio data. The generation unit analyzes the data input by the reception unit and generates subtitles. The detection unit detects comments that pose a risk of causing an uproar based on the subtitles generated by the generation unit. The editing unit inserts cuts or subtitles based on the comments detected by the detection unit. Effect of the Invention

[0007] The system according to the embodiment can analyze video and audio data, automatically detect comments that pose a risk of sparking outrage, and take appropriate action. [Brief description of the drawings]

[0008] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0010] First, the terms used in the following description will be explained.

[0011] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit).

[0012] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0013] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0014] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0016] [First embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0020] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 (see FIG. 2) acquires the data indicating the user input.

[0021] Output device 40 includes a display 40A and a speaker 40B, and presents data to a user by outputting the data in a form of expression that the user can perceive (e.g., voice and / or text). Display 40A displays visible information such as text and images according to instructions from processor 46. Speaker 40B outputs voice according to instructions from processor 46. Camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0022] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0023] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0024] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0025] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the emotion of the user using the emotion identification model 59 and perform identification processing using the user's emotion. In the emotion estimation function (emotion identification function) using the emotion identification model 59, various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, are performed, but are not limited to such examples. In addition, the estimation and prediction of the emotion also include, for example, analysis of the emotion.

[0026] In the smart device 14, the specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the smart device 14 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0027] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (prediction result, etc.) using the data generation model 58 by communicating with the server device having the data generation model 58. Also, the data processing device 12 may be a server device, or may be a terminal device (e.g., a mobile phone, a robot, a home appliance, etc.) owned by a user. Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example 1) The video and audio analysis system according to the embodiment of the present invention is a system that analyzes video and audio data and automatically generates and edits subtitles. This system inputs video and audio data, and the generation AI analyzes these data to automatically generate subtitles. The generation AI converts the voice into text using voice recognition technology and inserts subtitles according to the content of the video. Furthermore, the generation AI detects remarks that pose a risk of causing an uproar during analysis. For example, if an inappropriate expression or discriminatory remark is included, the generation AI detects it and inserts appropriate cuts or subtitles (such as annotations). This allows users to make video production more efficient and reduce the use of inappropriate expressions and the possibility of causing an uproar. For example, when a user uploads a video, the generation AI automatically analyzes the audio and generates subtitles. For example, in the case of an interview video, the remarks of the interviewer and the interviewee are displayed as subtitles. Furthermore, the generation AI analyzes the content of the remarks, and if inappropriate expressions are included, the generation AI cuts out the part or inserts a subtitle. For example, if a discriminatory remark is included, the generation AI cuts out the part and inserts a subtitle instead, such as "There was an inappropriate remark." This system not only allows users to significantly reduce the effort required for video production, but also reduces the risk of a backlash. For example, content creators can streamline video editing and provide appropriate content to viewers. Also, corporate public relations personnel can quickly create video content for internal and external use and avoid the risk of a backlash. In this way, the video and audio analysis system allows users to streamline video production and reduce the use of inappropriate language and the possibility of a backlash.

[0029] The video and audio analysis system according to the embodiment includes a reception unit, a generation unit, a detection unit, and an editing unit. The reception unit inputs video or audio data. The video or audio data includes, for example, a video file, an audio file, streaming data, and the like, but is not limited to such examples. The reception unit receives, for example, a video file uploaded by a user. The reception unit can also receive streaming data in real time. The generation unit uses a generation AI to analyze the data input by the reception unit and generate subtitles. The generation AI converts the voice into text using a voice recognition technology and inserts subtitles according to the content of the video. For example, the generation AI analyzes the voice using a deep learning-based voice recognition technology and generates text data. The generation AI can also improve the accuracy of the voice recognition using an acoustic model or a language model. The detection unit detects a statement that may cause a flaming incident based on the subtitles generated by the generation unit. The statement that may cause a flaming incident includes, for example, an inappropriate expression, a discriminatory statement, an offensive word, and the like, but is not limited to such examples. The detection unit detects an inappropriate expression using, for example, a specific keyword list. The detection unit can also use context analysis to understand the context of the remark and identify remarks that are at risk of causing a flaming incident. The editing unit inserts cuts and subtitles based on the remarks detected by the detection unit. The cuts and subtitle insertion include, for example, the range of the cut, the content and display position of the subtitle, and the like, but are not limited to such examples. The editing unit, for example, cuts the part of the detected inappropriate remark and inserts a subtitle such as "an inappropriate remark was made" instead. The editing unit can also insert appropriate annotations and explanations as subtitles according to the content of the detected remark. In this way, the video and audio analysis system according to the embodiment can analyze video and audio data and automatically generate and edit subtitles. A part or all of the above-mentioned processing in the generation unit may be performed, for example, using a generation AI, or may be performed without using a generation AI. For example, the generation unit can input audio data to the generation AI and generate text data using voice recognition technology.A part or all of the above-mentioned processing in the detection unit may be performed, for example, using AI, or may be performed without using AI. For example, the detection unit can input the generated subtitle data to AI and detect remarks that have a risk of causing an uproar. A part or all of the above-mentioned processing in the editing unit may be performed, for example, using AI, or may be performed without using AI. For example, the editing unit can input the detected remark data to AI and perform cuts or insertion of subtitles. In this way, the video and audio analysis system according to the embodiment can enable users to make video production more efficient and reduce the use of inappropriate expressions and the possibility of causing an uproar.

[0030] The reception unit inputs video or audio data. The video or audio data includes, but is not limited to, video files, audio files, streaming data, and the like. The reception unit, for example, receives video files uploaded by a user. The reception unit can also receive streaming data in real time. Specifically, the reception unit has a function of automatically detecting the file format and size when a user uploads a video file through a web interface and converting the file into an appropriate format. For example, the reception unit supports common video formats such as MP4, AVI, and MOV, and also supports audio file formats such as MP3, WAV, and AAC. When receiving streaming data, the data is received in real time and transmitted to the analysis unit while being buffered. This makes it possible to process data during live distribution without delay. Furthermore, the reception unit can automatically extract meta information of the data (for example, shooting date and time, location, device information, etc.) and use it for processing in the analysis unit and generation unit. This allows the reception unit to flexibly handle various data formats provided by users and improve the efficiency of the entire system.

[0031] The generation unit uses the generation AI to analyze the data input by the reception unit and generate subtitles. The generation AI converts the voice into text using voice recognition technology and inserts subtitles according to the content of the video. For example, the generation AI analyzes the voice using deep learning-based voice recognition technology and generates text data. The generation AI can also improve the accuracy of voice recognition using acoustic models and language models. Specifically, the generation AI first converts the voice data into acoustic features, and based on this, the voice recognition model identifies phonemes and words. Next, the language model generates optimal text taking into account the context. For example, the generation AI incorporates noise reduction technology to perform highly accurate voice recognition even in noisy environments. In addition, the generation AI can analyze the voice of each speaker individually using speaker separation technology and generate accurate subtitles even when multiple speakers are speaking at the same time. In addition, the generation AI automatically adjusts the display timing and position of the subtitles based on the content of the video. For example, by displaying subtitles according to important scenes in the video or the mouth of the speaker, it provides natural and easy-to-read subtitles for the viewer. This enables the generation unit to combine speech recognition technology and video analysis technology to automatically generate highly accurate, natural-sounding subtitles.

[0032] The detection unit detects statements that pose a risk of flaming based on the subtitles generated by the generation unit. Examples of statements that pose a risk of flaming include, but are not limited to, inappropriate expressions, discriminatory statements, and offensive words. The detection unit detects inappropriate expressions using, for example, a specific keyword list. The detection unit can also use context analysis to understand the context of a statement and identify statements that pose a risk of flaming. Specifically, the detection unit uses natural language processing (NLP) technology to analyze the subtitle text and detect inappropriate expressions based on the keyword list. Furthermore, to perform context analysis, a deep learning model for understanding the context is used to grasp the intention and nuance of the statement. For example, it is possible to identify expressions that have an offensive meaning according to the context, rather than simply matching keywords. The detection unit has the ability to learn past data and identify patterns of statements that pose a risk of flaming. This allows for rapid response even when new inappropriate expressions or offensive words appear. Furthermore, the detection unit can receive feedback from users and continuously improve the detection algorithm. This allows the detection unit to always respond to the latest risks and improve the reliability of the entire system.

[0033] The editing department inserts cuts and subtitles based on the remarks detected by the detection department. Examples of cuts and subtitle insertion include, but are not limited to, the range of cuts, the contents of subtitles, and the display position. For example, the editing department cuts the detected inappropriate remarks and inserts a subtitle such as "an inappropriate remark was made" instead. The editing department can also insert appropriate annotations and explanations as subtitles according to the contents of the detected remarks. Specifically, the editing department uses video editing software to automatically cut the detected inappropriate remarks and apply a transition effect to naturally connect the scenes before and after the cut. Furthermore, when inserting subtitles, fonts and colors that are easy for viewers to see are selected and displayed in an appropriate position. For example, if the contents of the subtitle are important, it is displayed large in the center of the screen, and if it is supplementary information, it is displayed small at the bottom of the screen. The editing department also provides an interface that allows the user to manually check the edited contents and correct them as necessary. This allows the user to easily check the automatically edited contents and adjust the final finish. Furthermore, the editing department saves the editing history and allows the user to recheck and re-edit the edited contents later. This allows the editing department to efficiently and flexibly edit video and audio data, and helps the user create high-quality content.

[0034] The generation unit can convert the voice into text using voice recognition technology and insert subtitles according to the content of the video. The generation unit can, for example, analyze the voice using deep learning-based voice recognition technology and generate text data. For example, the generation unit inputs voice data, extracts features of the voice using a deep learning model, and generates text data. The generation unit can also improve the accuracy of voice recognition using an acoustic model or a language model. For example, the generation unit analyzes the acoustic features of the voice using an acoustic model and understands the context of the voice using a language model. This allows the generation unit to convert the voice into text using voice recognition technology and insert subtitles according to the content of the video. A part or all of the above-mentioned processing in the generation unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the generation unit can input voice data to a generation AI and generate text data using voice recognition technology.

[0035] The detection unit can detect inappropriate expressions or discriminatory remarks. The detection unit detects inappropriate expressions, for example, using a specific keyword list. For example, the detection unit analyzes the voice data based on the keyword list and identifies a portion including an inappropriate expression. The detection unit can also understand the context of the remark using context analysis and identify a remark that is at risk of causing a flame war. For example, the detection unit analyzes the context of the voice data and detects inappropriate expressions based on the context. This allows the detection unit to automatically detect inappropriate expressions or discriminatory remarks. A part or all of the above-mentioned processing in the detection unit may be performed, for example, using AI, or may be performed without using AI. For example, the detection unit can input the voice data to AI and detect inappropriate expressions using context analysis.

[0036] The editorial department can make cuts or insert subtitles based on the detected remarks. For example, the editorial department cuts the part of the detected inappropriate remarks and inserts a subtitle such as "an inappropriate remark was made" instead. For example, the editorial department cuts a specific part of the audio data and inserts a subtitle. The editorial department can also insert appropriate annotations or explanations as subtitles depending on the content of the detected remarks. For example, if a discriminatory remark is included, the editorial department cuts that part and inserts a subtitle such as "an inappropriate remark was made" instead. This allows the editorial department to automatically insert appropriate cuts or subtitles based on the detected remarks. Some or all of the above-mentioned processing in the editorial department may be performed using, for example, AI, or may be performed without using AI. For example, the editorial department can input the detected remark data into AI and perform cuts or insert subtitles.

[0037] The generation unit can analyze a video uploaded by a user, and in the case of an interview video, display the remarks of the interviewer and the interviewee as subtitles. The generation unit can, for example, analyze a video uploaded by a user, and in the case of an interview video, display the remarks of the interviewer and the interviewee as subtitles. For example, the generation unit can analyze the audio data of the interview video and generate subtitles by distinguishing between the remarks of the interviewer and the interviewee. The generation unit can also analyze the video data of the interview video, perform face recognition of the speaker, and display subtitles corresponding to the speaker. This allows the generation unit to automatically display the remarks of the interview video as subtitles. A part or all of the above-mentioned processing in the generation unit may be performed, for example, using a generation AI, or may be performed without using a generation AI. For example, the generation unit can input the audio data of the interview video to the generation AI, convert the remarks into text data using voice recognition technology, and generate subtitles.

[0038] If a discriminatory remark is included, the editorial department can cut that part and insert a caption such as "There was an inappropriate remark" instead. For example, if a discriminatory remark is included, the editorial department can cut that part and insert a caption such as "There was an inappropriate remark" instead. For example, the editorial department cuts a specific part of the audio data and inserts a caption. The editorial department can also insert appropriate annotations or explanations as captions depending on the content of the detected remarks. For example, if a discriminatory remark is included, the editorial department cuts that part and inserts a caption such as "There was an inappropriate remark" instead. In this way, the editorial department can reduce the risk of a flame war by cutting the discriminatory remark and inserting an appropriate caption. Some or all of the above-mentioned processing in the editorial department may be performed using, for example, AI, or may be performed without using AI. For example, the editorial department can input the detected remark data into AI and perform cutting or inserting captions.

[0039] The reception unit can analyze the user's past data reception history and select the reception method. For example, the reception unit preferentially proposes a reception method (such as voice or text) that the user has frequently used in the past. For example, the reception unit analyzes the user's past data reception history and selects the optimal reception method. The reception unit can also predict and propose the optimal reception method for a specific time period from the user's past reception history. For example, the reception unit selects the optimal reception method for a specific time period based on the user's past reception history. The reception unit can also automatically select and provide an interface that the user has used in the past with preference. For example, the reception unit analyzes the user's past data reception history and selects the optimal interface. As a result, the reception unit can provide the user with the optimal reception method by analyzing the past data reception history. A part or all of the above-mentioned processing in the reception unit may be performed using, for example, AI, or may be performed without using AI. For example, the reception unit can input the user's past data reception history into AI and select the optimal reception method.

[0040] The reception unit can perform filtering based on the user's current project and field of interest when receiving video and audio data. The reception unit, for example, preferentially receives only data related to a project currently in progress by the user. For example, the reception unit refers to data of a project management tool and filters data related to the current project. The reception unit can also filter and receive highly relevant data based on the user's field of interest. For example, the reception unit analyzes the user's search history and browsing history and filters data based on the field of interest. The reception unit can also select and receive appropriate data based on topics in which the user has shown interest in the past. For example, the reception unit analyzes the user's past behavioral patterns and filters data based on the field of interest. As a result, the reception unit can preferentially receive highly relevant data by filtering data based on the user's project and field of interest. A part or all of the above-mentioned processing in the reception unit may be performed using, for example, AI, or may be performed without using AI. For example, the reception unit can input the user's project data and field of interest data into AI and perform filtering.

[0041] The reception unit can preferentially receive highly relevant data in consideration of the geographical location information of the user when receiving video and audio data. For example, when the user is in a specific area, the reception unit preferentially receives data related to the area. For example, the reception unit filters data related to the specific area based on the geographical location information of the user. In addition, when the user is traveling, the reception unit can also preferentially receive data related to the travel destination. For example, the reception unit filters data related to the travel destination based on the geographical location information of the user. In addition, when the user is at home, the reception unit can also preferentially receive information about the vicinity of the home. For example, the reception unit filters information about the vicinity of the home based on the geographical location information of the user. As a result, the reception unit can preferentially receive highly relevant data by considering the geographical location information of the user. A part or all of the above-mentioned processing in the reception unit may be performed using, for example, AI, or may be performed without using AI. For example, the reception unit can input the geographical location information of the user to AI and filter highly relevant data.

[0042] The reception unit can analyze the social media activity of the user and receive related data when receiving the video and audio data. The reception unit preferentially receives related data based on, for example, the content shared by the user on social media. For example, the reception unit analyzes the social media activity of the user and filters the related data. The reception unit can also receive related data based on the posted content of an account followed by the user on social media. For example, the reception unit analyzes the posted content of an account followed by the user and filters the related data. The reception unit can also receive related data based on a post that the user "liked" on social media. For example, the reception unit analyzes the "like" history of the user and filters the related data. In this way, the reception unit can analyze the social media activity of the user and preferentially receive data with high relevance. A part or all of the above-mentioned processing in the reception unit may be performed using, for example, AI, or may be performed without using AI. For example, the reception unit can input the social media activity data of the user to AI and filter the related data.

[0043] The generation unit can adjust the level of detail of the subtitles based on the importance of the voice when generating the subtitles. For example, in the case of an important statement, the generation unit generates a detailed subtitle to convey important information to the viewer. For example, the generation unit analyzes the voice data and evaluates the importance of the statement. In addition, in the case of a statement with low importance, the generation unit can generate a concise subtitle so as not to attract the viewer's attention. For example, the generation unit analyzes the tone and emphasis of the voice data and evaluates the importance of the statement. In addition, the generation unit can adjust the font size and color of the subtitles based on the tone and emphasis of the voice. For example, the generation unit adjusts the font size and color of the subtitles based on the tone and emphasis of the voice data. In this way, the generation unit can effectively convey important information to the viewer by adjusting the level of detail of the subtitles based on the importance of the voice. Part or all of the above-mentioned processing in the generation unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the generation unit can input the voice data to the generation AI, evaluate the importance of the voice, and adjust the level of detail of the subtitles.

[0044] The generation unit can apply different subtitle generation algorithms according to the category of the video when generating subtitles. For example, in the case of a documentary video, the generation unit generates subtitles using a formal font and color. For example, the generation unit analyzes the video data and identifies the category of the video. In addition, in the case of an entertainment video, the generation unit can also generate subtitles using a casual font and color. For example, the generation unit analyzes the content of the video data and identifies the category of the video. In addition, in the case of an educational video, the generation unit can also generate subtitles using a font and color with high visibility. For example, the generation unit analyzes the content of the video data and identifies the category of the video. In this way, the generation unit can provide subtitles that are easy for viewers to see by applying an appropriate subtitle generation algorithm according to the category of the video. Part or all of the above-mentioned processing in the generation unit may be performed, for example, using a generation AI, or may be performed without using a generation AI. For example, the generation unit can input video data to a generation AI, identify the category of the video, and apply an appropriate subtitle generation algorithm.

[0045] The generation unit can determine the priority of subtitles based on the time of recording of the audio when generating subtitles. The generation unit, for example, gives priority to the latest recorded data and provides the latest information to the viewer. For example, the generation unit determines the priority of subtitles based on the time of recording of the audio data. The generation unit can also postpone old recorded data and treat it as information of low importance. For example, the generation unit determines the priority of subtitles based on the time of recording of the audio data. The generation unit can also adjust the font size and color of the subtitles based on the time of recording. For example, the generation unit adjusts the font size and color of the subtitles based on the time of recording of the audio data. In this way, the generation unit can provide the latest information to the viewer by determining the priority of subtitles based on the time of recording of the audio. Part or all of the above-mentioned processing in the generation unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the generation unit can input audio data to the generation AI and determine the priority of subtitles based on the time of recording.

[0046] The generation unit can adjust the order of subtitles based on the relevance of the video when generating subtitles. The generation unit, for example, preferentially displays important subtitles in accordance with the main scenes of the video. For example, the generation unit analyzes the video data and identifies the main scenes. The generation unit can also display subtitles in a natural order according to the flow of the video. For example, the generation unit analyzes the content of the video data and displays the subtitles in a natural order. The generation unit can also arrange subtitles in an order that is easy for the viewer to understand based on the content of the video. For example, the generation unit analyzes the content of the video data and arranges subtitles in an order that is easy for the viewer to understand. In this way, the generation unit can provide subtitles in an order that is natural for the viewer by adjusting the order of subtitles based on the relevance of the video. A part or all of the above-mentioned processing in the generation unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the generation unit can input video data to the generation AI and adjust the order of subtitles based on the relevance of the video.

[0047] The detection unit can improve the accuracy of detection by considering the interrelationships of statements at the time of detection. The detection unit, for example, analyzes the context of statements and detects statements that have a risk of causing a flaming incident based on the context. For example, the detection unit analyzes the context of statements and identifies high-risk statements based on the context. The detection unit can also analyze interactions between multiple speakers and evaluate the risk based on the interrelationships. For example, the detection unit analyzes interactions between multiple speakers and identifies high-risk statements based on the interrelationships. The detection unit can also identify high-risk statements by considering the tone and emphasis of the statements. For example, the detection unit analyzes the tone and emphasis of the statements and identifies high-risk statements. As a result, the detection unit can improve the accuracy of detection by considering the interrelationships of statements. Part or all of the above-mentioned processing in the detection unit may be performed using, for example, AI, or may be performed without using AI. For example, the detection unit can input statement data to AI and analyze the interrelationships to identify high-risk statements.

[0048] The detection unit can perform detection while taking into consideration the attribute information of the speaker. The detection unit identifies high-risk statements based on, for example, the age and sex of the speaker. For example, the detection unit identifies high-risk statements based on the age and sex of the speaker. The detection unit can also identify high-risk statements based on the occupation and position of the speaker. For example, the detection unit identifies high-risk statements based on the occupation and position of the speaker. The detection unit can also identify high-risk statements by referring to the speaker's past speech history. For example, the detection unit identifies high-risk statements based on the speaker's past speech history. In this way, the detection unit can identify high-risk statements by taking into consideration the attribute information of the speaker. Part or all of the above-mentioned processing in the detection unit may be performed using, for example, AI, or may be performed without using AI. For example, the detection unit can input the attribute information of the speaker into AI and identify high-risk statements.

[0049] The detection unit can perform detection while taking into consideration the geographical distribution of the utterances. For example, the detection unit preferentially detects utterances in a specific region and evaluates the risk specific to the region. For example, the detection unit preferentially detects utterances in a specific region based on the geographical distribution of the utterances. The detection unit can also analyze the geographical distribution of the utterances to identify high-risk regions. For example, the detection unit identifies high-risk regions based on the geographical distribution of the utterances. The detection unit can also identify high-risk utterances based on the geographical distribution. For example, the detection unit identifies high-risk utterances based on the geographical distribution of the utterances. In this way, the detection unit can evaluate the risk specific to the region by taking into consideration the geographical distribution of the utterances. A part or all of the above-mentioned processing in the detection unit may be performed using, for example, AI, or may be performed without using AI. For example, the detection unit can input utterance data to AI and analyze the geographical distribution to identify high-risk utterances.

[0050] The detection unit can improve the accuracy of detection by referring to related literature of the utterance at the time of detection. The detection unit, for example, refers to literature related to the content of the utterance and identifies a high-risk utterance. For example, the detection unit refers to literature related to the content of the utterance and identifies a high-risk utterance. The detection unit can also evaluate the risk of the utterance based on the related literature. For example, the detection unit evaluates the risk of the utterance based on the related literature. The detection unit can also identify a high-risk utterance by referring to the related literature. For example, the detection unit refers to the related literature and identifies a high-risk utterance. As a result, the detection unit can identify a high-risk utterance by referring to the related literature. A part or all of the above-mentioned processing in the detection unit may be performed using, for example, AI, or may be performed without using AI. For example, the detection unit can input the utterance data to AI and identify a high-risk utterance by referring to the related literature.

[0051] The editing department can select the optimal editing method by analyzing the user's past editing history at the time of editing. For example, the editing department automatically applies an editing style that the user has used in the past. For example, the editing department analyzes the user's past editing history and selects the optimal editing method. The editing department can also extract a specific pattern from the user's past editing history and propose the optimal editing method. For example, the editing department extracts a specific pattern based on the user's past editing history and proposes the optimal editing method. The editing department can also perform editing by referring to the font and color of the subtitle used by the user in the past. For example, the editing department selects the font and color of the subtitle based on the user's past editing history. In this way, the editing department can provide the user with the optimal editing method by analyzing the past editing history. Part or all of the above-mentioned processing in the editing department may be performed using, for example, AI, or may be performed without using AI. For example, the editing department can input the user's past editing history into AI and select the optimal editing method.

[0052] The editing department can customize the editing means based on the user's current project at the time of editing. For example, the editing department preferentially provides editing means related to the user's ongoing project. For example, the editing department refers to the data of the project management tool and selects the editing means related to the current project. The editing department can also propose appropriate cuts and subtitles based on the theme of the user's project. For example, the editing department proposes appropriate cuts and subtitles based on the theme of the project. The editing department can also customize the editing means according to the progress of the user's project. For example, the editing department customizes the editing means according to the progress of the project. As a result, the editing department customizes the editing means based on the current project, thereby enabling optimal editing for the project. Part or all of the above-mentioned processing in the editing department may be performed, for example, using AI, or may be performed without using AI. For example, the editing department can input the user's project data into AI and select the optimal editing means.

[0053] The editing department can select the optimal editing method in consideration of the geographical location information of the user at the time of editing. For example, when the user is in a specific area, the editing department provides an editing means related to the area with priority. For example, the editing department selects an editing means related to a specific area based on the geographical location information of the user. In addition, when the user is traveling, the editing department can also provide an editing means related to the travel destination with priority. For example, the editing department selects an editing means related to the travel destination based on the geographical location information of the user. In addition, when the user is at home, the editing department can also select an editing means based on information around the home. For example, the editing department selects an editing means based on information around the home based on the geographical location information of the user. In this way, the editing department can provide the optimal editing method related to the area by considering the geographical location information. Part or all of the above-mentioned processing in the editing department may be performed using, for example, AI, or may be performed without using AI. For example, the editing department can input the geographical location information of the user into AI and select the optimal editing means.

[0054] The editorial department can analyze the user's social media activity and suggest a means of editing at the time of editing. The editorial department can suggest a relevant editing means based on, for example, the content shared by the user on social media. For example, the editorial department can analyze the user's social media activity and suggest a relevant editing means. The editorial department can also suggest a relevant editing means based on the posted content of an account the user follows on social media. For example, the editorial department can analyze the posted content of an account the user follows and suggest a relevant editing means. The editorial department can also suggest a relevant editing means based on a post that the user "liked" on social media. For example, the editorial department can analyze the user's "like" history and suggest a relevant editing means. In this way, the editorial department can suggest a relevant editing means by analyzing social media activity. Part or all of the above-mentioned processing in the editorial department may be performed, for example, using AI, or may be performed without using AI. For example, the editorial department can input the user's social media activity data into AI and suggest the optimal editing means.

[0055] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0056] The reception unit can analyze the user's past data reception history and select the optimal reception method. For example, it preferentially suggests reception methods (voice, text, etc.) that the user has used frequently in the past. The reception unit can also predict and suggest the optimal reception method for a specific time period from the user's past reception history. For example, it selects the optimal reception method for a specific time period based on the user's past reception history. The reception unit can also automatically select and provide an interface that the user has preferred to use in the past. In this way, the reception unit can provide the optimal reception method to the user by analyzing the past data reception history.

[0057] The generation unit can adjust the level of detail of the subtitles based on the importance of the audio when generating the subtitles. For example, in the case of an important statement, a detailed subtitle is generated to convey important information to the viewer. In addition, in the case of a statement of low importance, the generation unit can also generate a concise subtitle to avoid attracting the viewer's attention. Furthermore, the generation unit can adjust the font size and color of the subtitles based on the tone and emphasis of the audio. In this way, the generation unit can effectively convey important information to the viewer by adjusting the level of detail of the subtitles based on the importance of the audio.

[0058] The detection unit can improve the accuracy of detection by taking into account the interrelationships between statements during detection. For example, the detection unit analyzes the context of statements and detects statements that pose a risk of flaming based on the context. The detection unit can also analyze exchanges between multiple speakers and evaluate the risk based on the interrelationships. Furthermore, the detection unit can identify high-risk statements by taking into account the tone and emphasis of the statements. This allows the detection unit to improve the accuracy of detection by taking into account the interrelationships between statements.

[0059] When editing, the editing department can analyze the user's past editing history and select the optimal editing method. For example, the editing style that the user has used in the past as a favorite can be automatically applied. The editing department can also extract specific patterns from the user's past editing history and suggest the optimal editing method. Furthermore, the editing department can perform editing by referring to the font and color of subtitles that the user has used in the past. In this way, the editing department can provide the user with the optimal editing method by analyzing the past editing history.

[0060] When receiving video and audio data, the reception unit can preferentially receive highly relevant data by taking into consideration the geographical location information of the user. For example, when the user is in a specific area, data related to that area is preferentially received. In addition, when the user is traveling, the reception unit can also preferentially receive data related to the travel destination. Furthermore, when the user is at home, the reception unit can also preferentially receive information about the area around the user's home. In this way, the reception unit can preferentially receive highly relevant data by taking into consideration the geographical location information of the user.

[0061] The process flow of the first embodiment will be briefly described below.

[0062] Step 1: The reception unit inputs video or audio data. The video or audio data includes video files, audio files, streaming data, etc. The reception unit not only receives video files uploaded by users, but can also receive streaming data in real time. Step 2: The generation unit uses a generation AI to analyze the data input by the reception unit and generate subtitles. The generation AI converts the voice into text using voice recognition technology and inserts subtitles according to the content of the video. For example, the accuracy of voice recognition can be improved by using deep learning-based voice recognition technology, acoustic models, and language models. Step 3: The detection unit detects statements that may pose a risk of flaming based on the subtitles generated by the generation unit. Statements that may pose a risk of flaming include inappropriate expressions, discriminatory statements, and offensive words. The detection unit can detect inappropriate expressions using a specific keyword list, and can also use context analysis to understand the context of statements and identify statements that may pose a risk of flaming. Step 4: The editorial department inserts cuts or subtitles based on the remarks detected by the detection unit. The cuts and subtitle insertions include the range of the cut, the content of the subtitle, and the display position. For example, the part with the detected inappropriate remarks may be cut and a subtitle such as "An inappropriate remark was made" may be inserted instead. Appropriate annotations or explanations may also be inserted as subtitles depending on the content of the detected remarks.

[0063] (Example 2) The video and audio analysis system according to the embodiment of the present invention is a system that analyzes video and audio data and automatically generates and edits subtitles. This system inputs video and audio data, and the generation AI analyzes these data to automatically generate subtitles. The generation AI converts the voice into text using voice recognition technology and inserts subtitles according to the content of the video. Furthermore, the generation AI detects remarks that pose a risk of causing an uproar during analysis. For example, if an inappropriate expression or discriminatory remark is included, the generation AI detects it and inserts appropriate cuts or subtitles (such as annotations). This allows users to make video production more efficient and reduce the use of inappropriate expressions and the possibility of causing an uproar. For example, when a user uploads a video, the generation AI automatically analyzes the audio and generates subtitles. For example, in the case of an interview video, the remarks of the interviewer and the interviewee are displayed as subtitles. Furthermore, the generation AI analyzes the content of the remarks, and if inappropriate expressions are included, the generation AI cuts out the part or inserts a subtitle. For example, if a discriminatory remark is included, the generation AI cuts out the part and inserts a subtitle instead, such as "There was an inappropriate remark." This system not only allows users to significantly reduce the effort required for video production, but also reduces the risk of a backlash. For example, content creators can streamline video editing and provide appropriate content to viewers. Also, corporate public relations personnel can quickly create video content for internal and external use and avoid the risk of a backlash. In this way, the video and audio analysis system allows users to streamline video production and reduce the use of inappropriate language and the possibility of a backlash.

[0064] The video and audio analysis system according to the embodiment includes a reception unit, a generation unit, a detection unit, and an editing unit. The reception unit inputs video or audio data. The video or audio data includes, for example, a video file, an audio file, streaming data, and the like, but is not limited to such examples. The reception unit receives, for example, a video file uploaded by a user. The reception unit can also receive streaming data in real time. The generation unit uses a generation AI to analyze the data input by the reception unit and generate subtitles. The generation AI converts the voice into text using a voice recognition technology and inserts subtitles according to the content of the video. For example, the generation AI analyzes the voice using a deep learning-based voice recognition technology and generates text data. The generation AI can also improve the accuracy of the voice recognition using an acoustic model or a language model. The detection unit detects a statement that may cause a flaming incident based on the subtitles generated by the generation unit. The statement that may cause a flaming incident includes, for example, an inappropriate expression, a discriminatory statement, an offensive word, and the like, but is not limited to such examples. The detection unit detects an inappropriate expression using, for example, a specific keyword list. The detection unit can also use context analysis to understand the context of the remark and identify remarks that are at risk of causing a flaming incident. The editing unit inserts cuts and subtitles based on the remarks detected by the detection unit. The cuts and subtitle insertion include, for example, the range of the cut, the content and display position of the subtitle, and the like, but are not limited to such examples. The editing unit, for example, cuts the part of the detected inappropriate remark and inserts a subtitle such as "an inappropriate remark was made" instead. The editing unit can also insert appropriate annotations and explanations as subtitles according to the content of the detected remark. In this way, the video and audio analysis system according to the embodiment can analyze video and audio data and automatically generate and edit subtitles. A part or all of the above-mentioned processing in the generation unit may be performed, for example, using a generation AI, or may be performed without using a generation AI. For example, the generation unit can input audio data to the generation AI and generate text data using voice recognition technology.A part or all of the above-mentioned processing in the detection unit may be performed, for example, using AI, or may be performed without using AI. For example, the detection unit can input the generated subtitle data to AI and detect remarks that have a risk of causing an uproar. A part or all of the above-mentioned processing in the editing unit may be performed, for example, using AI, or may be performed without using AI. For example, the editing unit can input the detected remark data to AI and perform cuts or insertion of subtitles. In this way, the video and audio analysis system according to the embodiment can enable users to make video production more efficient and reduce the use of inappropriate expressions and the possibility of causing an uproar.

[0065] The reception unit inputs video or audio data. The video or audio data includes, but is not limited to, video files, audio files, streaming data, and the like. The reception unit, for example, receives video files uploaded by a user. The reception unit can also receive streaming data in real time. Specifically, the reception unit has a function of automatically detecting the file format and size when a user uploads a video file through a web interface and converting the file into an appropriate format. For example, the reception unit supports common video formats such as MP4, AVI, and MOV, and also supports audio file formats such as MP3, WAV, and AAC. When receiving streaming data, the data is received in real time and transmitted to the analysis unit while being buffered. This makes it possible to process data during live distribution without delay. Furthermore, the reception unit can automatically extract meta information of the data (for example, shooting date and time, location, device information, etc.) and use it for processing in the analysis unit and generation unit. This allows the reception unit to flexibly handle various data formats provided by users and improve the efficiency of the entire system.

[0066] The generation unit uses the generation AI to analyze the data input by the reception unit and generate subtitles. The generation AI converts the voice into text using voice recognition technology and inserts subtitles according to the content of the video. For example, the generation AI analyzes the voice using deep learning-based voice recognition technology and generates text data. The generation AI can also improve the accuracy of voice recognition using acoustic models and language models. Specifically, the generation AI first converts the voice data into acoustic features, and based on this, the voice recognition model identifies phonemes and words. Next, the language model generates optimal text taking into account the context. For example, the generation AI incorporates noise reduction technology to perform highly accurate voice recognition even in noisy environments. In addition, the generation AI can analyze the voice of each speaker individually using speaker separation technology and generate accurate subtitles even when multiple speakers are speaking at the same time. In addition, the generation AI automatically adjusts the display timing and position of the subtitles based on the content of the video. For example, by displaying subtitles according to important scenes in the video or the mouth of the speaker, it provides natural and easy-to-read subtitles for the viewer. This enables the generation unit to combine speech recognition technology and video analysis technology to automatically generate highly accurate, natural-sounding subtitles.

[0067] The detection unit detects statements that pose a risk of flaming based on the subtitles generated by the generation unit. Examples of statements that pose a risk of flaming include, but are not limited to, inappropriate expressions, discriminatory statements, and offensive words. The detection unit detects inappropriate expressions using, for example, a specific keyword list. The detection unit can also use context analysis to understand the context of a statement and identify statements that pose a risk of flaming. Specifically, the detection unit uses natural language processing (NLP) technology to analyze the subtitle text and detect inappropriate expressions based on the keyword list. Furthermore, to perform context analysis, a deep learning model for understanding the context is used to grasp the intention and nuance of the statement. For example, it is possible to identify expressions that have an offensive meaning according to the context, rather than simply matching keywords. The detection unit has the ability to learn past data and identify patterns of statements that pose a risk of flaming. This allows for rapid response even when new inappropriate expressions or offensive words appear. Furthermore, the detection unit can receive feedback from users and continuously improve the detection algorithm. This allows the detection unit to always respond to the latest risks and improve the reliability of the entire system.

[0068] The editing department inserts cuts and subtitles based on the remarks detected by the detection department. Examples of cuts and subtitle insertion include, but are not limited to, the range of cuts, the contents of subtitles, and the display position. For example, the editing department cuts the detected inappropriate remarks and inserts a subtitle such as "an inappropriate remark was made" instead. The editing department can also insert appropriate annotations and explanations as subtitles according to the contents of the detected remarks. Specifically, the editing department uses video editing software to automatically cut the detected inappropriate remarks and apply a transition effect to naturally connect the scenes before and after the cut. Furthermore, when inserting subtitles, fonts and colors that are easy for viewers to see are selected and displayed in an appropriate position. For example, if the contents of the subtitle are important, it is displayed large in the center of the screen, and if it is supplementary information, it is displayed small at the bottom of the screen. The editing department also provides an interface that allows the user to manually check the edited contents and correct them as necessary. This allows the user to easily check the automatically edited contents and adjust the final finish. Furthermore, the editing department saves the editing history and allows the user to recheck and re-edit the edited contents later. This allows the editing department to efficiently and flexibly edit video and audio data, and assists the user in producing high-quality content.

[0069] The generation unit can convert the voice into text using voice recognition technology and insert subtitles according to the content of the video. The generation unit can, for example, analyze the voice using deep learning-based voice recognition technology and generate text data. For example, the generation unit inputs voice data, extracts features of the voice using a deep learning model, and generates text data. The generation unit can also improve the accuracy of voice recognition using an acoustic model or a language model. For example, the generation unit analyzes the acoustic features of the voice using an acoustic model and understands the context of the voice using a language model. This allows the generation unit to convert the voice into text using voice recognition technology and insert subtitles according to the content of the video. A part or all of the above-mentioned processing in the generation unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the generation unit can input voice data to a generation AI and generate text data using voice recognition technology.

[0070] The detection unit can detect inappropriate expressions or discriminatory remarks. The detection unit detects inappropriate expressions, for example, using a specific keyword list. For example, the detection unit analyzes the voice data based on the keyword list and identifies a portion including an inappropriate expression. The detection unit can also understand the context of the remark using context analysis and identify a remark that is at risk of causing a flame war. For example, the detection unit analyzes the context of the voice data and detects inappropriate expressions based on the context. This allows the detection unit to automatically detect inappropriate expressions or discriminatory remarks. A part or all of the above-mentioned processing in the detection unit may be performed, for example, using AI, or may be performed without using AI. For example, the detection unit can input the voice data to AI and detect inappropriate expressions using context analysis.

[0071] The editorial department can make cuts or insert subtitles based on the detected remarks. For example, the editorial department cuts the part of the detected inappropriate remarks and inserts a subtitle such as "an inappropriate remark was made" instead. For example, the editorial department cuts a specific part of the audio data and inserts a subtitle. The editorial department can also insert appropriate annotations or explanations as subtitles depending on the content of the detected remarks. For example, if a discriminatory remark is included, the editorial department cuts that part and inserts a subtitle such as "an inappropriate remark was made" instead. This allows the editorial department to automatically insert appropriate cuts or subtitles based on the detected remarks. Some or all of the above-mentioned processing in the editorial department may be performed using, for example, AI, or may be performed without using AI. For example, the editorial department can input the detected remark data into AI and perform cuts or insert subtitles.

[0072] The generation unit can analyze a video uploaded by a user, and in the case of an interview video, display the remarks of the interviewer and the interviewee as subtitles. The generation unit can, for example, analyze a video uploaded by a user, and in the case of an interview video, display the remarks of the interviewer and the interviewee as subtitles. For example, the generation unit can analyze the audio data of the interview video and generate subtitles by distinguishing between the remarks of the interviewer and the interviewee. The generation unit can also analyze the video data of the interview video, perform face recognition of the speaker, and display subtitles corresponding to the speaker. This allows the generation unit to automatically display the remarks of the interview video as subtitles. Part or all of the above-mentioned processing in the generation unit may be performed, for example, using a generation AI, or may be performed without using a generation AI. For example, the generation unit can input the audio data of the interview video to the generation AI, convert the remarks into text data using voice recognition technology, and generate subtitles.

[0073] If a discriminatory remark is included, the editorial department can cut that part and insert a caption such as "There was an inappropriate remark" instead. For example, if a discriminatory remark is included, the editorial department can cut that part and insert a caption such as "There was an inappropriate remark" instead. For example, the editorial department cuts a specific part of the audio data and inserts a caption. The editorial department can also insert appropriate annotations or explanations as captions depending on the content of the detected remarks. For example, if a discriminatory remark is included, the editorial department cuts that part and inserts a caption such as "There was an inappropriate remark" instead. In this way, the editorial department can reduce the risk of a flame war by cutting the discriminatory remark and inserting an appropriate caption. Some or all of the above-mentioned processing in the editorial department may be performed using, for example, AI, or may be performed without using AI. For example, the editorial department can input the detected remark data into AI and perform cutting or inserting captions.

[0074] The reception unit can estimate the user's emotion and adjust the timing of receiving the video and audio data based on the estimated user's emotion. For example, when the user is feeling stressed, the reception unit delays the timing of receiving the video and audio data and waits until the user is relaxed. For example, the reception unit captures the user's facial expression with a camera and estimates the user's emotion using an emotion estimation algorithm. For example, the reception unit calculates an emotion score based on the change in facial expression and evaluates the user's emotional state. In addition, when the user is relaxed, the reception unit can immediately receive the video and audio data and provide a smooth operation. For example, the reception unit records the user's voice and estimates the emotion using a voice analysis technique. For example, the reception unit analyzes the tone and speed of the voice and calculates the emotion score. In addition, when the user is in a hurry, the reception unit can advance the timing of receiving the data and process the data quickly. For example, the reception unit collects the user's biometric data (heart rate and electrodermal activity) with a sensor and estimates the emotion using an emotion estimation algorithm. For example, the reception unit calculates an emotion score based on the fluctuation of the heart rate and evaluates the user's emotional state. As a result, the reception unit can reduce the user's stress by adjusting the reception timing according to the user's emotions. The emotion estimation is realized by using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the reception unit may be performed using, for example, an AI, or may be performed without using an AI. For example, the reception unit can input the user's emotion data to the generation AI and cause the generation AI to execute emotion estimation.

[0075] The reception unit can analyze the user's past data reception history and select the reception method. For example, the reception unit preferentially proposes a reception method (such as voice or text) that the user has frequently used in the past. For example, the reception unit analyzes the user's past data reception history and selects the optimal reception method. The reception unit can also predict and propose the optimal reception method for a specific time period from the user's past reception history. For example, the reception unit selects the optimal reception method for a specific time period based on the user's past reception history. The reception unit can also automatically select and provide an interface that the user has used in the past with preference. For example, the reception unit analyzes the user's past data reception history and selects the optimal interface. As a result, the reception unit can provide the user with the optimal reception method by analyzing the past data reception history. A part or all of the above-mentioned processing in the reception unit may be performed using, for example, AI, or may be performed without using AI. For example, the reception unit can input the user's past data reception history into AI and select the optimal reception method.

[0076] The reception unit can perform filtering based on the user's current project and field of interest when receiving video and audio data. The reception unit, for example, preferentially receives only data related to a project currently in progress by the user. For example, the reception unit refers to data of a project management tool and filters data related to the current project. The reception unit can also filter and receive highly relevant data based on the user's field of interest. For example, the reception unit analyzes the user's search history and browsing history and filters data based on the field of interest. The reception unit can also select and receive appropriate data based on topics in which the user has shown interest in the past. For example, the reception unit analyzes the user's past behavioral patterns and filters data based on the field of interest. As a result, the reception unit can preferentially receive highly relevant data by filtering data based on the user's project and field of interest. A part or all of the above-mentioned processing in the reception unit may be performed using, for example, AI, or may be performed without using AI. For example, the reception unit can input the user's project data and field of interest data into AI and perform filtering.

[0077] The reception unit can estimate the user's emotion and determine the priority of data to be received based on the estimated user's emotion. For example, when the user is feeling stressed, the reception unit postpones data of low importance and preferentially receives data of high importance. For example, the reception unit captures the user's facial expression with a camera and estimates the user's emotion using an emotion estimation algorithm. For example, the reception unit calculates an emotion score based on changes in facial expression and evaluates the user's emotional state. In addition, when the user is relaxed, the reception unit can equally receive all data. For example, the reception unit records the user's voice and estimates the emotion using a voice analysis technique. For example, the reception unit analyzes the tone and speed of the voice and calculates the emotion score. In addition, when the user is in a hurry, the reception unit can preferentially receive the most important data and process it quickly. For example, the reception unit collects the user's biometric data (heart rate and electrodermal activity) with a sensor and estimates the emotion using an emotion estimation algorithm. For example, the reception unit calculates an emotion score based on fluctuations in heart rate and evaluates the user's emotional state. This allows the reception unit to prioritize data according to the user's emotions, thereby allowing important data to be preferentially received. The emotion estimation is realized using an emotion estimation function, for example, using an emotion engine or a generation AI. The generation AI is a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to such examples. Some or all of the above-mentioned processing in the reception unit may be performed, for example, using an AI, or may be performed without using an AI. For example, the reception unit can input the user's emotion data to the generation AI and cause the generation AI to execute emotion estimation.

[0078] The reception unit can preferentially receive highly relevant data in consideration of the geographical location information of the user when receiving video and audio data. For example, when the user is in a specific area, the reception unit preferentially receives data related to the area. For example, the reception unit filters data related to the specific area based on the geographical location information of the user. In addition, when the user is traveling, the reception unit can also preferentially receive data related to the travel destination. For example, the reception unit filters data related to the travel destination based on the geographical location information of the user. In addition, when the user is at home, the reception unit can also preferentially receive information about the vicinity of the home. For example, the reception unit filters information about the vicinity of the home based on the geographical location information of the user. As a result, the reception unit can preferentially receive highly relevant data by considering the geographical location information of the user. A part or all of the above-mentioned processing in the reception unit may be performed using, for example, AI, or may be performed without using AI. For example, the reception unit can input the geographical location information of the user to AI and filter highly relevant data.

[0079] The reception unit can analyze the social media activity of the user and receive related data when receiving the video and audio data. The reception unit preferentially receives related data based on, for example, the content shared by the user on social media. For example, the reception unit analyzes the social media activity of the user and filters the related data. The reception unit can also receive related data based on the posted content of an account followed by the user on social media. For example, the reception unit analyzes the posted content of an account followed by the user and filters the related data. The reception unit can also receive related data based on a post that the user "liked" on social media. For example, the reception unit analyzes the "like" history of the user and filters the related data. In this way, the reception unit can analyze the social media activity of the user and preferentially receive data with high relevance. A part or all of the above-mentioned processing in the reception unit may be performed using, for example, AI, or may be performed without using AI. For example, the reception unit can input the social media activity data of the user to AI and filter the related data.

[0080] The generation unit can estimate the user's emotion and adjust the expression method of the subtitles based on the estimated user's emotion. For example, when the user is relaxed, the generation unit generates subtitles using soft fonts and colors. For example, the generation unit captures the user's facial expression with a camera and estimates the user's emotion using an emotion estimation algorithm. For example, the generation unit calculates an emotion score based on the change in facial expression and evaluates the user's emotional state. In addition, when the user is nervous, the generation unit can also generate subtitles using a simple, highly visible font. For example, the generation unit records the user's voice and estimates the emotion using a voice analysis technology. For example, the generation unit analyzes the tone and speed of the voice and calculates the emotion score. In addition, when the user is excited, the generation unit can also generate subtitles with a visually stimulating effect added. For example, the generation unit collects the user's biometric data (heart rate and electrodermal activity) with a sensor and estimates the emotion using an emotion estimation algorithm. For example, the generation unit calculates an emotion score based on the fluctuation of the heart rate and evaluates the user's emotional state. As a result, the generation unit can provide subtitles that are easy for viewers to read by adjusting the way the subtitles are expressed according to the user's emotions. The emotion estimation is realized using an emotion estimation function, for example, using an emotion engine or a generation AI. The generation AI is a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to such examples. Some or all of the above-mentioned processing in the generation unit may be performed, for example, using AI, or may be performed without using AI. For example, the generation unit can input user emotion data to the generation AI and cause the generation AI to execute emotion estimation.

[0081] The generation unit can adjust the level of detail of the subtitles based on the importance of the voice when generating the subtitles. For example, in the case of an important statement, the generation unit generates a detailed subtitle to convey important information to the viewer. For example, the generation unit analyzes the voice data and evaluates the importance of the statement. In addition, in the case of a statement with low importance, the generation unit can generate a concise subtitle so as not to attract the viewer's attention. For example, the generation unit analyzes the tone and emphasis of the voice data and evaluates the importance of the statement. In addition, the generation unit can adjust the font size and color of the subtitles based on the tone and emphasis of the voice. For example, the generation unit adjusts the font size and color of the subtitles based on the tone and emphasis of the voice data. In this way, the generation unit can effectively convey important information to the viewer by adjusting the level of detail of the subtitles based on the importance of the voice. Part or all of the above-mentioned processing in the generation unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the generation unit can input the voice data to the generation AI, evaluate the importance of the voice, and adjust the level of detail of the subtitles.

[0082] The generation unit can apply different subtitle generation algorithms according to the category of the video when generating subtitles. For example, in the case of a documentary video, the generation unit generates subtitles using a formal font and color. For example, the generation unit analyzes the video data and identifies the category of the video. In addition, the generation unit can also generate subtitles using a casual font and color in the case of an entertainment video. For example, the generation unit analyzes the content of the video data and identifies the category of the video. In addition, the generation unit can generate subtitles using a font and color with high visibility in the case of an educational video. For example, the generation unit analyzes the content of the video data and identifies the category of the video. In this way, the generation unit can provide subtitles that are easy for viewers to see by applying an appropriate subtitle generation algorithm according to the category of the video. Part or all of the above-mentioned processing in the generation unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the generation unit can input video data to a generation AI, identify the category of the video, and apply an appropriate subtitle generation algorithm.

[0083] The generating unit can estimate the user's emotion and adjust the length of the subtitle based on the estimated user's emotion. For example, when the user is in a hurry, the generating unit generates a short subtitle that is to the point. For example, the generating unit captures the user's facial expression with a camera and estimates the user's emotion using an emotion estimation algorithm. For example, the generating unit calculates an emotion score based on the change in facial expression and evaluates the user's emotional state. In addition, when the user is relaxed, the generating unit can generate a longer subtitle that includes a detailed explanation. For example, the generating unit records the user's voice and estimates the emotion using a voice analysis technique. For example, the generating unit analyzes the tone and speed of the voice and calculates the emotion score. In addition, when the user is excited, the generating unit can generate a subtitle with a visually stimulating effect. For example, the generating unit collects the user's biometric data (heart rate and electrodermal activity) with a sensor and estimates the emotion using an emotion estimation algorithm. For example, the generating unit calculates an emotion score based on the fluctuation of the heart rate and evaluates the user's emotional state. As a result, the generation unit can provide subtitles that are easy for viewers to view by adjusting the length of the subtitles according to the user's emotions. The emotion estimation is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to such examples. Some or all of the above-mentioned processing in the generation unit may be performed using, for example, an AI, or may be performed without using an AI. For example, the generation unit can input the user's emotion data to the generation AI and cause the generation AI to execute emotion estimation.

[0084] The generation unit can determine the priority of subtitles based on the time of recording of the audio when generating subtitles. The generation unit, for example, gives priority to the latest recorded data and provides the latest information to the viewer. For example, the generation unit determines the priority of subtitles based on the time of recording of the audio data. The generation unit can also postpone old recorded data and treat it as information of low importance. For example, the generation unit determines the priority of subtitles based on the time of recording of the audio data. The generation unit can also adjust the font size and color of the subtitles based on the time of recording. For example, the generation unit adjusts the font size and color of the subtitles based on the time of recording of the audio data. In this way, the generation unit can provide the latest information to the viewer by determining the priority of subtitles based on the time of recording of the audio. Part or all of the above-mentioned processing in the generation unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the generation unit can input audio data to the generation AI and determine the priority of subtitles based on the time of recording.

[0085] The generation unit can adjust the order of subtitles based on the relevance of the video when generating subtitles. The generation unit, for example, preferentially displays important subtitles in accordance with the main scenes of the video. For example, the generation unit analyzes the video data and identifies the main scenes. The generation unit can also display subtitles in a natural order according to the flow of the video. For example, the generation unit analyzes the content of the video data and displays the subtitles in a natural order. The generation unit can also arrange subtitles in an order that is easy for the viewer to understand based on the content of the video. For example, the generation unit analyzes the content of the video data and arranges subtitles in an order that is easy for the viewer to understand. In this way, the generation unit can provide subtitles in an order that is natural for the viewer by adjusting the order of subtitles based on the relevance of the video. A part or all of the above-mentioned processing in the generation unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the generation unit can input video data to the generation AI and adjust the order of subtitles based on the relevance of the video.

[0086] The detection unit can estimate the user's emotion and adjust the detection criteria for statements that may cause a flaming risk based on the estimated user's emotion. For example, when the user is nervous, the detection unit detects statements that may cause a flaming risk using strict criteria. For example, the detection unit captures the user's facial expression with a camera and estimates the user's emotion using an emotion estimation algorithm. For example, the detection unit calculates an emotion score based on changes in facial expression and evaluates the user's emotional state. In addition, when the user is relaxed, the detection unit can also detect statements that may cause a flaming risk using flexible criteria. For example, the detection unit records the user's voice and estimates the emotion using a voice analysis technique. For example, the detection unit analyzes the tone and speed of the voice and calculates the emotion score. In addition, when the user is excited, the detection unit can generate subtitles with visually stimulating effects. For example, the detection unit collects the user's biometric data (heart rate and electrodermal activity) using a sensor and estimates the emotion using an emotion estimation algorithm. For example, the detection unit calculates an emotion score based on fluctuations in heart rate and evaluates the user's emotional state. This allows the detection unit to effectively detect comments that pose a risk of causing a flame war by adjusting the detection criteria according to the user's emotions. The emotion estimation is realized using an emotion estimation function, for example, using an emotion engine or a generation AI. The generation AI is a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to such examples. Some or all of the above-mentioned processing in the detection unit may be performed, for example, using AI, or may be performed without using AI. For example, the detection unit can input the user's emotion data to the generation AI and cause the generation AI to perform emotion estimation.

[0087] The detection unit can improve the accuracy of detection by considering the interrelationships of statements at the time of detection. The detection unit, for example, analyzes the context of statements and detects statements that have a risk of causing a flaming incident based on the context. For example, the detection unit analyzes the context of statements and identifies high-risk statements based on the context. The detection unit can also analyze interactions between multiple speakers and evaluate the risk based on the interrelationships. For example, the detection unit analyzes interactions between multiple speakers and identifies high-risk statements based on the interrelationships. The detection unit can also identify high-risk statements by considering the tone and emphasis of the statements. For example, the detection unit analyzes the tone and emphasis of the statements and identifies high-risk statements. As a result, the detection unit can improve the accuracy of detection by considering the interrelationships of statements. Part or all of the above-mentioned processing in the detection unit may be performed using, for example, AI, or may be performed without using AI. For example, the detection unit can input statement data to AI and analyze the interrelationships to identify high-risk statements.

[0088] The detection unit can perform detection while taking into consideration the attribute information of the speaker. The detection unit identifies high-risk statements based on, for example, the age and sex of the speaker. For example, the detection unit identifies high-risk statements based on the age and sex of the speaker. The detection unit can also identify high-risk statements based on the occupation and position of the speaker. For example, the detection unit identifies high-risk statements based on the occupation and position of the speaker. The detection unit can also identify high-risk statements by referring to the speaker's past speech history. For example, the detection unit identifies high-risk statements based on the speaker's past speech history. In this way, the detection unit can identify high-risk statements by taking into consideration the attribute information of the speaker. Part or all of the above-mentioned processing in the detection unit may be performed using, for example, AI, or may be performed without using AI. For example, the detection unit can input the attribute information of the speaker into AI and identify high-risk statements.

[0089] The detection unit can estimate the user's emotion and adjust the display order of the detection results based on the estimated user's emotion. For example, when the user is nervous, the detection unit preferentially displays important detection results. For example, the detection unit captures the user's facial expression with a camera and estimates the user's emotion using an emotion estimation algorithm. For example, the detection unit calculates an emotion score based on the change in facial expression and evaluates the user's emotional state. In addition, when the user is relaxed, the detection unit can equally display all detection results. For example, the detection unit records the user's voice and estimates the emotion using a voice analysis technique. For example, the detection unit analyzes the tone and speed of the voice and calculates the emotion score. In addition, when the user is in a hurry, the detection unit can preferentially display the most important detection results. For example, the detection unit collects the user's biometric data (heart rate and electrodermal activity) with a sensor and estimates the emotion using an emotion estimation algorithm. For example, the detection unit calculates an emotion score based on the fluctuation of the heart rate and evaluates the user's emotional state. This allows the detection unit to adjust the display order of the detection results according to the user's emotions, thereby enabling important information to be preferentially displayed. The emotion estimation is realized using an emotion estimation function, for example, using an emotion engine or a generation AI. The generation AI is a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to such examples. Some or all of the above-mentioned processing in the detection unit may be performed, for example, using an AI, or may be performed without using an AI. For example, the detection unit can input the user's emotion data to the generation AI and cause the generation AI to execute emotion estimation.

[0090] The detection unit can perform detection while taking into consideration the geographical distribution of the utterances. For example, the detection unit preferentially detects utterances in a specific region and evaluates the risk specific to the region. For example, the detection unit preferentially detects utterances in a specific region based on the geographical distribution of the utterances. The detection unit can also analyze the geographical distribution of the utterances to identify high-risk regions. For example, the detection unit identifies high-risk regions based on the geographical distribution of the utterances. The detection unit can also identify high-risk utterances based on the geographical distribution. For example, the detection unit identifies high-risk utterances based on the geographical distribution of the utterances. In this way, the detection unit can evaluate the risk specific to the region by taking into consideration the geographical distribution of the utterances. A part or all of the above-mentioned processing in the detection unit may be performed using, for example, AI, or may be performed without using AI. For example, the detection unit can input utterance data to AI and analyze the geographical distribution to identify high-risk utterances.

[0091] The detection unit can improve the accuracy of detection by referring to related literature of the utterance at the time of detection. The detection unit, for example, refers to literature related to the content of the utterance and identifies a high-risk utterance. For example, the detection unit refers to literature related to the content of the utterance and identifies a high-risk utterance. The detection unit can also evaluate the risk of the utterance based on the related literature. For example, the detection unit evaluates the risk of the utterance based on the related literature. The detection unit can also identify a high-risk utterance by referring to the related literature. For example, the detection unit refers to the related literature and identifies a high-risk utterance. As a result, the detection unit can identify a high-risk utterance by referring to the related literature. A part or all of the above-mentioned processing in the detection unit may be performed using, for example, AI, or may be performed without using AI. For example, the detection unit can input the utterance data to AI and identify a high-risk utterance by referring to the related literature.

[0092] The editing department can estimate the user's emotion and adjust the method of inserting cuts and subtitles based on the estimated user's emotion. For example, when the user is nervous, the editing department inserts a simple, highly visible subtitle. For example, the editing department captures the user's facial expression with a camera and estimates the user's emotion using an emotion estimation algorithm. For example, the editing department calculates an emotion score based on the change in facial expression and evaluates the user's emotional state. In addition, when the user is relaxed, the editing department can also insert a visually attractive subtitle. For example, the editing department records the user's voice and estimates the emotion using a voice analysis technology. For example, the editing department analyzes the tone and speed of the voice and calculates the emotion score. In addition, when the user is in a hurry, the editing department can quickly make cuts and insert only important information as a subtitle. For example, the editing department collects the user's biometric data (heart rate and electrodermal activity) with a sensor and estimates the emotion using an emotion estimation algorithm. For example, the editing department calculates an emotion score based on the fluctuation in heart rate and evaluates the user's emotional state. This allows the editing department to adjust the method of inserting cuts and subtitles according to the user's emotions, thereby enabling editing that is easy for viewers to view. The emotion estimation is realized using an emotion estimation function, for example, using an emotion engine or a generation AI. The generation AI is a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. Some or all of the above-mentioned processing in the editing department may be performed, for example, using AI, or may be performed without using AI. For example, the editing department can input the user's emotion data into the generation AI and have the generation AI execute emotion estimation.

[0093] The editing department can select the optimal editing method by analyzing the user's past editing history at the time of editing. For example, the editing department automatically applies an editing style that the user has used in the past. For example, the editing department analyzes the user's past editing history and selects the optimal editing method. The editing department can also extract a specific pattern from the user's past editing history and propose the optimal editing method. For example, the editing department extracts a specific pattern based on the user's past editing history and proposes the optimal editing method. The editing department can also perform editing by referring to the font and color of the subtitle used by the user in the past. For example, the editing department selects the font and color of the subtitle based on the user's past editing history. In this way, the editing department can provide the user with the optimal editing method by analyzing the past editing history. Part or all of the above-mentioned processing in the editing department may be performed using, for example, AI, or may be performed without using AI. For example, the editing department can input the user's past editing history into AI and select the optimal editing method.

[0094] The editing department can customize the editing means based on the user's current project at the time of editing. For example, the editing department preferentially provides editing means related to the user's ongoing project. For example, the editing department refers to the data of the project management tool and selects the editing means related to the current project. The editing department can also propose appropriate cuts and subtitles based on the theme of the user's project. For example, the editing department proposes appropriate cuts and subtitles based on the theme of the project. The editing department can also customize the editing means according to the progress of the user's project. For example, the editing department customizes the editing means according to the progress of the project. As a result, the editing department customizes the editing means based on the current project, thereby enabling optimal editing for the project. Part or all of the above-mentioned processing in the editing department may be performed, for example, using AI, or may be performed without using AI. For example, the editing department can input the user's project data into AI and select the optimal editing means.

[0095] The editing department can estimate the user's emotion and determine the priority of cuts and subtitles based on the estimated user's emotion. For example, when the user is nervous, the editing department inserts important cuts and subtitles with priority. For example, the editing department captures the user's facial expression with a camera and estimates the user's emotion using an emotion estimation algorithm. For example, the editing department calculates an emotion score based on the change in facial expression and evaluates the user's emotional state. In addition, when the user is relaxed, the editing department can insert all cuts and subtitles evenly. For example, the editing department records the user's voice and estimates the emotion using a voice analysis technology. For example, the editing department analyzes the tone and speed of the voice and calculates the emotion score. In addition, when the user is in a hurry, the editing department can insert the most important cuts and subtitles with priority. For example, the editing department collects the user's biometric data (heart rate and electrodermal activity) with a sensor and estimates the emotion using an emotion estimation algorithm. For example, the editing department calculates an emotion score based on the fluctuation of the heart rate and evaluates the user's emotional state. This allows the editing department to prioritize important information by determining the priority of cuts and subtitles according to the user's emotions. The emotion estimation is realized by using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. Some or all of the above-mentioned processing in the editing department may be performed using, for example, an AI, or may be performed without using an AI. For example, the editing department can input the user's emotion data into the generation AI and have the generation AI execute emotion estimation.

[0096] The editing department can select the optimal editing method in consideration of the geographical location information of the user at the time of editing. For example, when the user is in a specific area, the editing department provides an editing means related to the area with priority. For example, the editing department selects an editing means related to a specific area based on the geographical location information of the user. In addition, when the user is traveling, the editing department can also provide an editing means related to the travel destination with priority. For example, the editing department selects an editing means related to the travel destination based on the geographical location information of the user. In addition, when the user is at home, the editing department can also select an editing means based on information around the home. For example, the editing department selects an editing means based on information around the home based on the geographical location information of the user. In this way, the editing department can provide the optimal editing method related to the area by considering the geographical location information. Part or all of the above-mentioned processing in the editing department may be performed using, for example, AI, or may be performed without using AI. For example, the editing department can input the geographical location information of the user into AI and select the optimal editing means.

[0097] The editorial department can analyze the user's social media activity and suggest a means of editing at the time of editing. The editorial department can suggest a relevant editing means based on, for example, the content shared by the user on social media. For example, the editorial department can analyze the user's social media activity and suggest a relevant editing means. The editorial department can also suggest a relevant editing means based on the posted content of an account the user follows on social media. For example, the editorial department can analyze the posted content of an account the user follows and suggest a relevant editing means. The editorial department can also suggest a relevant editing means based on a post that the user "liked" on social media. For example, the editorial department can analyze the user's "like" history and suggest a relevant editing means. In this way, the editorial department can suggest a relevant editing means by analyzing social media activity. Part or all of the above-mentioned processing in the editorial department may be performed, for example, using AI, or may be performed without using AI. For example, the editorial department can input the user's social media activity data into AI and suggest the optimal editing means.

[0098] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0099] The reception unit can analyze the user's past data reception history and select the optimal reception method. For example, it preferentially suggests reception methods (voice, text, etc.) that the user has used frequently in the past. The reception unit can also predict and suggest the optimal reception method for a specific time period from the user's past reception history. For example, it selects the optimal reception method for a specific time period based on the user's past reception history. The reception unit can also automatically select and provide an interface that the user has preferred to use in the past. In this way, the reception unit can provide the optimal reception method to the user by analyzing the past data reception history.

[0100] The generation unit can adjust the level of detail of the subtitles based on the importance of the audio when generating the subtitles. For example, in the case of an important statement, a detailed subtitle is generated to convey important information to the viewer. In addition, in the case of a statement of low importance, the generation unit can also generate a concise subtitle to avoid attracting the viewer's attention. Furthermore, the generation unit can adjust the font size and color of the subtitles based on the tone and emphasis of the audio. In this way, the generation unit can effectively convey important information to the viewer by adjusting the level of detail of the subtitles based on the importance of the audio.

[0101] The detection unit can improve the accuracy of detection by taking into account the interrelationships between statements during detection. For example, the detection unit analyzes the context of statements and detects statements that pose a risk of flaming based on the context. The detection unit can also analyze exchanges between multiple speakers and evaluate the risk based on the interrelationships. Furthermore, the detection unit can identify high-risk statements by taking into account the tone and emphasis of the statements. This allows the detection unit to improve the accuracy of detection by taking into account the interrelationships between statements.

[0102] When editing, the editing department can analyze the user's past editing history and select the optimal editing method. For example, the editing style that the user has used in the past as a favorite can be automatically applied. The editing department can also extract specific patterns from the user's past editing history and suggest the optimal editing method. Furthermore, the editing department can perform editing by referring to the font and color of subtitles that the user has used in the past. In this way, the editing department can provide the user with the optimal editing method by analyzing the past editing history.

[0103] When receiving video and audio data, the reception unit can preferentially receive highly relevant data by taking into consideration the geographical location information of the user. For example, when the user is in a specific area, data related to that area is preferentially received. In addition, when the user is traveling, the reception unit can also preferentially receive data related to the travel destination. Furthermore, when the user is at home, the reception unit can also preferentially receive information about the area around the user's home. In this way, the reception unit can preferentially receive highly relevant data by taking into consideration the geographical location information of the user.

[0104] The reception unit can estimate the user's emotions and adjust the timing of receiving video and audio data based on the estimated user's emotions. For example, if the user is feeling stressed, the reception timing can be delayed to wait until the user is relaxed. Also, if the user is relaxed, the reception unit can immediately receive video and audio data to provide smooth operation. Furthermore, if the user is in a hurry, the reception unit can advance the reception timing to quickly process the data. In this way, the reception unit can reduce the user's stress by adjusting the reception timing according to the user's emotions.

[0105] The generation unit can estimate the user's emotion and adjust the subtitle expression method based on the estimated user's emotion. For example, if the user is relaxed, the generation unit can generate subtitles using soft fonts and colors. In addition, if the user is nervous, the generation unit can generate subtitles using simple, highly visible fonts. Furthermore, if the user is excited, the generation unit can generate subtitles with visually stimulating effects added. In this way, the generation unit can provide subtitles that are easy for viewers to see by adjusting the subtitle expression method according to the user's emotion.

[0106] The detection unit can estimate the user's emotions and adjust the detection criteria for statements that pose a risk of igniting a flame war based on the estimated user's emotions. For example, when the user is nervous, the detection unit detects statements that pose a risk of igniting a flame war using strict criteria. Also, when the user is relaxed, the detection unit can detect statements that pose a risk of igniting a flame war using flexible criteria. Furthermore, when the user is excited, the detection unit can generate subtitles that add visually stimulating effects. In this way, the detection unit can effectively detect statements that pose a risk of igniting a flame war by adjusting the detection criteria according to the user's emotions.

[0107] The editing department can estimate the user's emotions and adjust the method of inserting cuts and subtitles based on the estimated user's emotions. For example, if the user is nervous, the editing department can insert a simple, highly visible subtitle. Also, if the user is relaxed, the editing department can insert a visually appealing subtitle. Furthermore, if the user is in a hurry, the editing department can make quick cuts and insert only important information as subtitles. In this way, the editing department can adjust the method of inserting cuts and subtitles according to the user's emotions, making it possible to edit in a way that is easy for viewers to view.

[0108] The reception unit can estimate the user's emotions and determine the priority order of data to be received based on the estimated user's emotions. For example, when the user is feeling stressed, data with low importance is postponed and data with high importance is received preferentially. In addition, when the user is relaxed, the reception unit can receive all data equally. Furthermore, when the user is in a hurry, the reception unit can receive the most important data preferentially and process it quickly. In this way, the reception unit can receive important data preferentially by determining the priority order of data according to the user's emotions.

[0109] The process flow of the second embodiment will be briefly described below.

[0110] Step 1: The reception unit inputs video or audio data. The video or audio data includes video files, audio files, streaming data, etc. The reception unit not only receives video files uploaded by users, but can also receive streaming data in real time. Step 2: The generation unit uses a generation AI to analyze the data input by the reception unit and generate subtitles. The generation AI converts the voice into text using voice recognition technology and inserts subtitles according to the content of the video. For example, the accuracy of voice recognition can be improved by using deep learning-based voice recognition technology, acoustic models, and language models. Step 3: The detection unit detects statements that may pose a risk of flaming based on the subtitles generated by the generation unit. Statements that may pose a risk of flaming include inappropriate expressions, discriminatory statements, and offensive words. The detection unit can detect inappropriate expressions using a specific keyword list, and can also use context analysis to understand the context of statements and identify statements that may pose a risk of flaming. Step 4: The editorial department inserts cuts or subtitles based on the remarks detected by the detection unit. The cuts and subtitle insertions include the range of the cut, the content of the subtitle, and the display position. For example, the part with the detected inappropriate remarks may be cut and a subtitle such as "An inappropriate remark was made" may be inserted instead. Appropriate annotations or explanations may also be inserted as subtitles depending on the content of the detected remarks.

[0111] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0112] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image (e.g., still image data or video data) are input. The data generation model 58 infers the input inference data according to the instruction indicated by the prompt, and outputs the inference result in one or more data formats of voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing while using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction, and in this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. In the data processing device 12 and the like, the data generation model 58 includes a plurality of types, and the data generation model 58 includes an AI other than the generation AI. The AI ​​other than the generation AI is, for example, a linear regression, a logistic regression, a decision tree, a random forest, a support vector machine (SVM), a k-means clustering, a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), or a naive Bayes, and can perform various processes, but is not limited to such examples. The AI ​​may also be an AI agent. In addition, when the processes of each of the above-mentioned parts are performed by an AI, the processes are performed in part or in whole by the AI, but are not limited to such examples.In addition, processing performed by AI including the generation AI may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by AI including the generation AI.

[0113] Furthermore, the above-described processing by the data processing system 10 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information required for processing from the data processing device 12 or an external device.

[0114] Each of the multiple elements including the above-mentioned reception unit, generation unit, detection unit, and editing unit is realized, for example, by at least one of the smart device 14 and the data processing device 12. For example, the reception unit is realized by the control unit 46A of the smart device 14 and receives a video file uploaded by a user. The generation unit is realized, for example, by the specific processing unit 290 of the data processing device 12 and analyzes audio using a generation AI to generate subtitles. The detection unit is realized, for example, by the specific processing unit 290 of the data processing device 12 and analyzes the generated subtitle data to detect remarks that may cause a flame war. The editing unit is realized, for example, by the control unit 46A of the smart device 14 and inserts cuts and subtitles based on the detected remarks. The correspondence between each unit and the device or control unit is not limited to the above-mentioned example, and various changes are possible.

[0115] [Second embodiment] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0116] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0117] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0118] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0119] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0120] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0121] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0122] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0123] The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0124] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the emotion of the user using the emotion identification model 59 and perform identification processing using the user's emotion. In the emotion estimation function (emotion identification function) using the emotion identification model 59, various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, are performed, but are not limited to such examples. In addition, the estimation and prediction of the emotion also include, for example, analysis of the emotion.

[0125] In the smart glasses 214, the specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the smart glasses 214 have a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0126] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (such as a prediction result) using the data generation model 58 by communicating with the server device having the data generation model 58. In addition, the data processing device 12 may be a server device, or may be a terminal device (such as a mobile phone, a robot, or a home appliance) owned by a user.

[0127] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0128] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image (e.g., still image data or video data) is input. The data generation model 58 infers the input inference data according to the instruction indicated by the prompt, and outputs the inference result in one or more data formats of voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing while using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction, and in this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. In the data processing device 12 and the like, the data generation model 58 includes a plurality of types, and the data generation model 58 includes an AI other than the generation AI. The AI ​​other than the generation AI is, for example, a linear regression, a logistic regression, a decision tree, a random forest, a support vector machine (SVM), a k-means clustering, a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), or a naive Bayes, and can perform various processes, but is not limited to such examples. The AI ​​may be an AI agent. In addition, when the processes of each part described above are performed by an AI, the processes are performed in part or in whole by the AI, but are not limited to such examples. In addition, the processes performed by the AI ​​including the generation AI may be replaced with rule-based processes, and the rule-based processes may be replaced with processes performed by the AI ​​including the generation AI.

[0129] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but may be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or an external device, and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or an external device.

[0130] Each of the multiple elements including the above-mentioned reception unit, generation unit, detection unit, and editing unit is realized, for example, by at least one of the smart glasses 214 and the data processing device 12. For example, the reception unit is realized by the control unit 46A of the smart glasses 214 and receives a video file uploaded by a user. The generation unit is realized, for example, by the specific processing unit 290 of the data processing device 12 and analyzes audio using a generation AI to generate subtitles. The detection unit is realized, for example, by the specific processing unit 290 of the data processing device 12 and analyzes the generated subtitle data to detect remarks that may cause a flame war. The editing unit is realized, for example, by the control unit 46A of the smart glasses 214 and inserts cuts and subtitles based on the detected remarks. The correspondence between each unit and the device or control unit is not limited to the above-mentioned example, and various changes are possible.

[0131] [Third embodiment] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0132] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0133] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0134] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0135] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0136] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0137] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0138] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0139] The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0140] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the emotion of the user using the emotion identification model 59 and perform identification processing using the user's emotion. In the emotion estimation function (emotion identification function) using the emotion identification model 59, various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, are performed, but are not limited to such examples. In addition, the estimation and prediction of the emotion also include, for example, analysis of the emotion.

[0141] In the headset type terminal 314, the specific processing is performed by the processor 46. The storage 50 stores a specific program 60. The processor 46 reads the specific program 60 from the storage 50, and executes the read specific program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific program 60 executed on the RAM 48. Note that the headset type terminal 314 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the identification processing unit 290 using these models.

[0142] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (such as a prediction result) using the data generation model 58 by communicating with the server device having the data generation model 58. In addition, the data processing device 12 may be a server device, or may be a terminal device (such as a mobile phone, a robot, or a home appliance) owned by a user.

[0143] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0144] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image (e.g., still image data or video data) is input. The data generation model 58 infers the input inference data according to the instruction indicated by the prompt, and outputs the inference result in one or more data formats of voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing while using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction, and in this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. In the data processing device 12 and the like, the data generation model 58 includes a plurality of types, and the data generation model 58 includes an AI other than the generation AI. The AI ​​other than the generation AI is, for example, a linear regression, a logistic regression, a decision tree, a random forest, a support vector machine (SVM), a k-means clustering, a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), or a naive Bayes, and can perform various processes, but is not limited to such examples. The AI ​​may be an AI agent. In addition, when the processes of each part described above are performed by an AI, the processes are performed in part or in whole by the AI, but are not limited to such examples. In addition, the processes performed by the AI ​​including the generation AI may be replaced with rule-based processes, and the rule-based processes may be replaced with processes performed by the AI ​​including the generation AI.

[0145] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset type terminal 314, but may be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset type terminal 314. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the headset type terminal 314 or an external device, and the headset type terminal 314 acquires or collects information required for processing from the data processing device 12 or an external device.

[0146] Each of the multiple elements including the above-mentioned reception unit, generation unit, detection unit, and editing unit is realized, for example, by at least one of the headset type terminal 314 and the data processing device 12. For example, the reception unit is realized by the control unit 46A of the headset type terminal 314, and receives a video file uploaded by a user. The generation unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and analyzes audio using a generation AI to generate subtitles. The detection unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and analyzes the generated subtitle data to detect remarks that may cause a flame war. The editing unit is realized, for example, by the control unit 46A of the headset type terminal 314, and inserts cuts and subtitles based on the detected remarks. The correspondence between each unit and the device or control unit is not limited to the above-mentioned example, and various changes are possible.

[0147] [Fourth embodiment] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0148] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0149] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0150] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. In addition, the microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0151] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0152] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS image sensor or a CCD image sensor, and captures images of the user's surroundings (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of a typical healthy person).

[0153] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0154] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and legs. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, and legs. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0155] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0156] The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0157] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the emotion of the user using the emotion identification model 59 and perform identification processing using the user's emotion. In the emotion estimation function (emotion identification function) using the emotion identification model 59, various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, are performed, but are not limited to such examples. In addition, the estimation and prediction of the emotion also include, for example, analysis of the emotion.

[0158] In the robot 414, the specific processing is performed by the processor 46. The storage 50 stores a specific program 60. The processor 46 reads out the specific program 60 from the storage 50, and executes the specific program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific program 60 executed on the RAM 48. The robot 414 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0159] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (such as a prediction result) using the data generation model 58 by communicating with the server device having the data generation model 58. In addition, the data processing device 12 may be a server device, or may be a terminal device (such as a mobile phone, a robot, or a home appliance) owned by a user.

[0160] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0161] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image (e.g., still image data or video data) is input. The data generation model 58 infers the input inference data according to the instruction indicated by the prompt, and outputs the inference result in one or more data formats of voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing while using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction, and in this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. In the data processing device 12 and the like, the data generation model 58 includes a plurality of types, and the data generation model 58 includes an AI other than the generation AI. The AI ​​other than the generation AI is, for example, a linear regression, a logistic regression, a decision tree, a random forest, a support vector machine (SVM), a k-means clustering, a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), or a naive Bayes, and can perform various processes, but is not limited to such examples. The AI ​​may be an AI agent. In addition, when the processes of each part described above are performed by an AI, the processes are performed in part or in whole by the AI, but are not limited to such examples. In addition, the processes performed by the AI ​​including the generation AI may be replaced with rule-based processes, and the rule-based processes may be replaced with processes performed by the AI ​​including the generation AI.

[0162] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but may be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or an external device, etc., and the robot 414 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0163] Each of the multiple elements including the above-mentioned reception unit, generation unit, detection unit, and editing unit is realized, for example, by at least one of the robot 414 and the data processing device 12. For example, the reception unit is realized by the control unit 46A of the robot 414, and receives a video file uploaded by a user. The generation unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and analyzes audio using a generation AI to generate subtitles. The detection unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and analyzes the generated subtitle data to detect remarks that may cause a flaming risk. The editing unit is realized, for example, by the control unit 46A of the robot 414, and inserts cuts and subtitles based on the detected remarks. The correspondence between each unit and the device or control unit is not limited to the above-mentioned example, and various changes are possible.

[0164] The emotion identification model 59 as an emotion engine may determine the emotion of the user according to a specific mapping. Specifically, the emotion identification model 59 may determine the emotion of the user according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the emotion of the robot, and the identification processing unit 290 may perform identification processing using the emotion of the robot.

[0165] FIG. 9 is a diagram showing an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged. The more outside the concentric circles, the more emotions that represent states and actions that arise from a state of mind are arranged. Emotions are a concept that includes emotions and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions that occur in the brain are arranged. On the right side of the concentric circles, emotions that are generally induced by situational judgment are arranged. On the upper and lower sides of the concentric circles, emotions that are generally generated from reactions that occur in the brain and are induced by situational judgment are arranged. In addition, on the upper side of the concentric circles, emotions of "pleasure" are arranged, and on the lower side, emotions of "discomfort" are arranged. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0166] These emotions are distributed in the three o'clock direction of emotion map 400 and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0167] The inside of emotion map 400 represents what is going on inside one's mind, and the outside of emotion map 400 represents behavior, so the further out you go on emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[0168] Here, human emotions are based on various balances such as posture and blood sugar level, and when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. Emotions can also be created for robots, cars, motorcycles, etc., based on various balances such as posture and battery level, so that when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. The emotion map may be generated, for example, based on the emotion map of Dr. Mitsuyoshi (Research on speech emotion recognition and emotion brain physiological signal analysis system, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). On the left half of the emotion map, emotions belonging to an area called "reaction" where sensation is dominant are lined up. On the right half of the emotion map, emotions belonging to an area called "situation" where situation recognition is dominant are lined up.

[0169] The emotion map defines two emotions that promote learning. The first is the negative emotion around the middle of "repentance" or "remorse" on the situation side. In other words, this is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the positive emotion around "desire" on the response side. In other words, this is when the robot has positive feelings such as "I want more" or "I want to know more."

[0170] The emotion identification model 59 inputs the user input to a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the emotion of the user. This neural network is pre-trained based on multiple learning data that are combinations of the user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in Fig. 10. Fig. 10 shows an example in which multiple emotions, "relief," "calm," and "encouraging," have similar emotion values.

[0171] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers, including computer 22.

[0172] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable computer-readable non-transitory storage medium such as a Universal Serial Bus (USB) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0173] In addition, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 upon request from the data processing device 12.

[0174] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0175] As the hardware resource for executing the specific process, various processors as shown below can be used. An example of the processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the specific process by executing software, i.e., a program. Another example of the processor is a dedicated electric circuit, which is a processor having a circuit configuration designed exclusively for executing a specific process, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), or an Application Specific Integrated Circuit (ASIC). Each processor has a built-in or connected memory, and each processor executes the specific process by using the memory.

[0176] The hardware resource that executes the specific process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[0177] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a configuration using a processor that realizes the functions of the entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0178] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. The specific processes described above are merely examples. It goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processes may be changed without departing from the spirit of the invention.

[0179] In the above example, the first to fourth embodiments are described separately, but some or all of these embodiments may be combined. The smart device 14, the smart glasses 214, the headset terminal 314, and the robot 414 are only examples, and they may be combined with each other, or may be other devices. In the above example, the first and second embodiments are described separately, but they may be combined.

[0180] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replaced with respect to the description and illustrations shown above, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the description and illustrations shown above omit explanations of technical common sense that do not require particular explanation in enabling the implementation of the technology of the present disclosure.

[0181] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, and standard was specifically and individually indicated to be incorporated by reference.

[0182] (Appendix 1) a reception unit for inputting video or audio data; a generation unit that analyzes the data input by the reception unit and generates subtitles; A detection unit that detects comments that pose a risk of causing an uproar based on the subtitles generated by the generation unit; and an editing unit that cuts or inserts a subtitle based on the remark detected by the detection unit. A system characterized by: (Appendix 2) The generation unit is Uses voice recognition technology to convert speech into text and inserts subtitles according to the content of the video. 2. The system of claim 1, (Appendix 3) The system described in Appendix 1, wherein the detection unit detects inappropriate expressions or discriminatory remarks. (Appendix 4) The editorial department: Insert cuts and subtitles based on detected utterances 2. The system of claim 1, (Appendix 5) The generation unit is Analyzes videos uploaded by users, and in the case of interview videos, displays the statements of the interviewer and interviewee as subtitles. 2. The system of claim 1, (Appendix 6) The editorial department: If a video contains discriminatory remarks, cut out that part and replace it with a caption saying, "Inappropriate remarks were made." 2. The system of claim 1, (Appendix 7) The reception unit is To estimate a user's emotion and adjust the timing of receiving video and audio data based on the estimated user's emotion. 2. The system of claim 1, (Appendix 8) The system according to claim 1, wherein the reception unit analyzes the user's past data reception history and selects a reception method. (Appendix 9) The reception unit is Filtering video and audio data as it arrives based on the user's current projects and interests 2. The system of claim 1, (Appendix 10) The reception unit is Estimate user emotions and prioritize data to be accepted based on the estimated user emotions. 2. The system of claim 1, (Appendix 11) The reception unit is When accepting video and audio data, the system takes into account the user's geographic location information and prioritizes accepting data that is highly relevant. 2. The system of claim 1, (Appendix 12) The reception unit is When receiving video and audio data, analyze the user's social media activity and receive related data. 2. The system of claim 1, (Appendix 13) The generation unit is Estimate the user's emotions and adjust the way subtitles are presented based on the estimated user's emotions 2. The system of claim 1, (Appendix 14) The generation unit is When generating subtitles, adjust the level of detail in the subtitles based on the importance of the audio 2. The system of claim 1, (Appendix 15) The generation unit is When generating subtitles, different subtitle generation algorithms are applied depending on the video category. 2. The system of claim 1, (Appendix 16) The generation unit is Estimate user emotions and adjust the length of subtitles based on the estimated user emotions 2. The system of claim 1, (Appendix 17) The generation unit is When generating subtitles, prioritize subtitles based on when the audio was recorded 2. The system of claim 1, (Appendix 18) The generation unit is When generating subtitles, adjust the order of subtitles based on video relevance 2. The system of claim 1, (Appendix 19) The detection unit is Estimate user sentiment and adjust the detection criteria for comments that may cause a flaming incident based on the estimated user sentiment. 2. The system of claim 1, (Appendix 20) The detection unit is When detecting, the accuracy of detection is improved by taking into account the correlation of statements. 2. The system of claim 1, (Appendix 21) The detection unit is When detecting, the speaker's attribute information is taken into consideration 2. The system of claim 1, (Appendix 22) The detection unit is Estimate the user's emotions and adjust the display order of the detection results based on the estimated user emotions. 2. The system of claim 1, (Appendix 23) The detection unit is When detecting, the geographic distribution of comments is taken into account. 2. The system of claim 1, (Appendix 24) The detection unit is When detecting, the accuracy of detection is improved by referencing related literature of the utterance. 2. The system of claim 1, (Appendix 25) The editorial department: Estimate the user's emotions and adjust the way cuts and subtitles are inserted based on the estimated user's emotions. 2. The system of claim 1, (Appendix 26) The editorial department: When editing, the system analyzes the user's past editing history and selects the optimal editing method. 2. The system of claim 1, (Appendix 27) The editorial department: When editing, customize the editing experience based on the user's current project 2. The system of claim 1, (Appendix 28) The editorial department: Estimate the user's emotions and prioritize cuts and subtitles based on the estimated user's emotions. 2. The system of claim 1, (Appendix 29) The editorial department: When editing, the geographic location of the user is taken into account to select the best editing method 2. The system of claim 1, (Appendix 30) The editorial department: When editing, the app analyzes users' social media activity to suggest editing strategies 2. The system of claim 1, [Explanation of symbols]

[0183] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot

Claims

1. a reception unit for inputting video or audio data; a generation unit that analyzes the data input by the reception unit and generates subtitles; A detection unit that detects comments that pose a risk of causing an uproar based on the subtitles generated by the generation unit; and an editing unit that cuts or inserts a subtitle based on the remark detected by the detection unit. A system characterized by:

2. The generation unit is Uses voice recognition technology to convert speech into text and inserts subtitles according to the content of the video.

2. The system of claim 1 .

3. The detection unit is Detect inappropriate or discriminatory language 2. The system of claim 1 .

4. The generation unit is Analyzes videos uploaded by users, and in the case of interview videos, displays the statements of the interviewer and interviewee as subtitles.

2. The system of claim 1 .

5. The editorial department: If a video contains discriminatory remarks, cut out the part and replace it with a caption indicating that the remarks are inappropriate.

2. The system of claim 1 .

6. The reception unit is To estimate a user's emotion and adjust the timing of receiving video and audio data based on the estimated user's emotion.

2. The system of claim 1 .

7. The reception unit is Analyze the user's past data reception history and select the reception method 2. The system of claim 1 .

Images